Phase 1: hypothesis generation

The aim of phase 1 was to develop a study-specific logic model identifying the factors that we hypothesised would influence successful implementation of REACT. This model was informed by implementation theory and later served as a guide to data collection and analysis, making explicit our underlying assumptions and ensuring that analytic generalisations based on our findings could inform the growing field of implementation science.72

We operationalised our model as a series of propositions in the form of:

X will happen, if Y.

For this study, this became:

The implementation of REACT is more likely to be successful, if . . . [list of key factors].

Most implementation theories are ‘mid-range theories’, defined as useful frameworks to guide practitioners in a general sense.88 They may tell us the kind of factors likely to be important, but do not identify specific factors in any one context. For this, we need to generate a programme-specific logic model that integrates mid-range formal theories with informal theories (i.e. ideas and learning based on the previous evidence and experience of key stakeholders).

Developing the IMPART study logic model

Our logic model was informed by four key elements:

• NPT73
• a systematic review of relevant implementation studies of DHIs for people with psychosis or bipolar disorder and/or their relatives81
• qualitative analysis of data from the REACT feasibility trial,53 including interviews with EIP staff and relatives
• SRGs, including staff and relatives at each participating trust.

Synthesis of the findings from each element was informed by the clinical and theoretical expertise of our research team. The process is shown in Figure 4.

FIGURE 4 Generation of study-specific hypotheses and study logic model. (Figure not reproduced here; see the original article.)

Our preliminary work, prior to the grant, had led us to hypothesise that uptake and use of REACT by relatives would be best encouraged by embedding REACT into routine health care, thus ensuring that relatives were introduced to the intervention by a trusted professional, who would subsequently promote its ongoing use. NPT suggested that this embedding into routine health care would be itself facilitated by positive feedback from relatives (reflexive monitoring, Figure 5). Thus, although our primary outcomes related to uptake and use by health-care professionals, we were also interested in uptake and use by relatives.

FIGURE 5 Preliminary logic model at the time when the grant was submitted. (Figure not reproduced here; see the original article.)

Developing the IMPART study-specific hypotheses

The methods and findings for each of the four pieces of work in phase 1 that informed the study-specific hypotheses are published in full elsewhere (NPT73 and the literature review81) or as appendices to this report (see Appendix 2 for the secondary analysis and Appendix 3 for the SRG report). In this chapter, we focus on the key findings from each and explain how these were synthesised to form the IMPART study-specific hypotheses. Appendix 4 shows how the key findings from each element in phase 1 informed the final study-specific hypotheses and evaluates these hypotheses in the light of the study’s findings.

Normalisation process theory

Normalisation process theory was used to inform the structure of the framework into which our hypotheses were organised, and which then guided our data collection and analysis. The framework was used flexibly and, in phase 2, evolved to include additional constructs. However, in phase 1, we used the 16 core components, organised within the four core constructs of NPT as outlined on the NPT toolkit:89

1. coherence – the sense-making work people do individually and collectively when faced with operationalising a set of practices
2. cognitive participation – the relational work that people do to build and sustain a community of practice around a new technology or intervention
3. collective action – the operational work people do to enact a set of practices
4. reflexive monitoring – the appraisal work people do to assess and understand how the new intervention affects them and others.

We used the online toolkit to train the whole team to ensure that everyone had a good understanding of the theory. Although our team included internationally recognised experts in the field of implementation science (notably JR-M and EM), it was important that all team members, including carer researchers, and other methodological and clinical experts also had a thorough working knowledge of the theory so that everyone could contribute to the data analysis.

We then used this shared understanding to map the findings from our literature review, secondary analysis of REACT feasibility study data and SRG meetings into the framework. This helped to develop the generic mid-range NPT components into study-specific hypotheses for the IMPART study. The process for this is presented in Appendix 4, Tables 27–31.

As an example, the NPT component ‘relational integration’ in the construct ‘collective action’ hypothesises that an intervention is more likely to succeed if staff maintain their trust in each other’s work and expertise throughout the intervention. Data from all three studies in phase 1 contributed to making this a study-specific hypothesis in REACT:

REACT is more likely to be successfully implemented (more relatives are invited to use it) if REACT is offered as part of an integrated package of care for relatives, and not a stand-alone intervention; access is restricted to relatives already supported within the team; staff can see and are confident that the forum is well managed; staff can see and are confident that direct messages are being responded to; REACT supporters and IMPART leads are clear and confident about managing risk identified in REACT.

Systematic literature review

We conducted a systematic review of studies published between January 1995 and October 2017 that identified factors affecting implementation of DHIs for people with psychosis or bipolar disorder, or their relatives.81 In this chapter, we focus on the key factors relevant to understanding the work undertaken by staff to implement DHIs, and the uptake and use by service users or relatives. The review identified 26 eligible papers describing DHIs for service users, but none aimed at supporting relatives. This further highlighted the need for the IMPART study.

The majority of factors for effective implementation of DHIs were focused on the characteristics of staff supporting the DHIs, or the design of the intervention. The first key finding was that users were more likely to engage with a DHI if it was facilitated by a staff member or peer supporter, and had been proposed by a staff member who could understand and clearly articulate its benefits. Enthusiastic staff can serve as champions for DHIs, reinforcing engagement among service users and providing practical guidance. They can also mitigate the experience of DHIs as generic and impersonal. The importance of human support in successful implementation was a strong theme across several studies, and is consistent with the findings of other systematic reviews.90

Engagement among service users was facilitated by the DHIs’ accessibility (independently, in their own time and in their own home) and the ability to share information with friends and family. Some felt that the DHIs made communication with staff members easier on issues such as drug and alcohol use, which they would have found difficult to broach face to face, or relieved the boredom of time spent in inpatient facilities.

Key barriers to engagement for both staff and service users were concerns about cybersecurity, complexity and poor interoperability. Concerns about data privacy or security were associated with disengagement. DHIs that were difficult to use or time-consuming did not succeed, and DHIs that did not integrate with existing IT systems were found to be frustrating, either because they required additional logins or because important information was not being shared between platforms. DHIs that were considered either too complex for users or too simplistic were also less likely to be sustained by staff or service users.

Some service users expressed concerns about the potentially negative impact of a DHI on their mental health, with reports of increases in paranoia or other exacerbations of symptoms. Symptom severity, particularly episodes of acute ill health, and negative symptoms of schizophrenia were barriers to engagement. These factors highlighted the importance of user-centred design in all aspects, and including all users.

Finally, cost was a consistent factor in the review. Although digital interventions are promoted as offering potential long-term savings, insufficient funding in the short term was linked to inadequate staff time, training, space and necessary equipment, all of which negatively affected implementation.

These factors were mapped onto the relevant NPT framework constructs.

Secondary analysis of REACT feasibility study data

The aim of this secondary analysis was to identify factors affecting implementation of REACT (in its paper version) during the REACT feasibility trial (2008–12).53 This was a stratified RCT in which 103 relatives were allocated to receive either treatment as usual (TAU) or TAU plus the paper-based REACT toolkit intervention, in a ratio of 1 : 1. Compared with TAU only, those receiving the intervention showed reduced distress and perceived increased support and ability to cope at 6-month follow-up.

Participants were relatives, partners or close friends of people experiencing psychosis who were supported by EIP services in three participating trusts. Each participant was offered an initial face-to-face session introducing the toolkit and agreeing arrangements for support, which was offered by e-mail or telephone (relatives’ preference) for up to 1 hour per week over 6 months, with a minimum of monthly contact.

Six REACT supporters working across the EIP teams were trained and offered monthly group supervision for the duration of the project. The first 14 relatives to complete the 6-month follow-up were invited to take part in a qualitative interview about their experiences of REACT. Twelve consented but one data file was corrupted and could not be analysed. Interviews were also conducted with the four supporters still involved in the study at the end.

Our secondary analysis of these data sought to identify specific factors affecting implementation of REACT. Despite the toolkit being mainly offered in paper form during the trial [an online Portable Document Format (PDF) was also available], we still felt that the data had important insights to offer into the process of implementing this self-management toolkit for relatives as part of an existing care pathway. See Appendix 2 for a detailed description of the methods, participants and findings. Appendix 2, Table 25, provides a summary of the main findings.

The key themes to emerge from our analysis were the importance of timing (when REACT is offered); the perceived benefits for the relative; that structure and delivery of the support; and the balancing of the REACT supporter role with other job demands.

Timing

Both relatives and supporters identified the need for REACT to be offered to relatives as early as possible to ensure that it was available as they were making sense of the challenges they faced and developing ways of coping. It should therefore be clearly identified as an early step in the care pathway. However, an early offer should be regularly followed up with prompts to explore the use of REACT and highlight its potential value to relatives:

Everybody I met loved it, there was not criticism for the toolkit, except for ‘I’ve already figured this information out for myself, but had this been [there] a year ago . . . [it] would have been a godsend’.

[My supporter] used to ring me and I would say, oh, something trivial was bothering me; he said, well why don’t you look at section this, section that, and I shouldn’t have to be prompted but, he would and then I would start reading it again then you know, . . . I was really yes, into it then.

Perceived benefits for relatives

REACT supporters were motivated to offer the intervention to relatives when they saw it as being directly relevant to the diverse range of relatives they worked with, easily accessible, engaging and up to date. Their work was driven by wanting to see specific benefits for relatives, including feeling valued in their role and developing greater knowledge about mental health and services, which would lead to greater confidence to engage with services and indirectly improve their relationship with service users:

It does really benefit if people understand what they are dealing with – I think that’s half the battle . . . [otherwise] they are just fighting against the unknown really all the time.

Relatives highlighted the use of case studies as particularly engaging as they realised that they were not alone in the problems they faced. Even those who felt that they would have benefited from earlier access to REACT valued the reassurance of information that was consistent with the understanding they had already developed:

It confirmed what you had worked out for yourself, that you were actually on the right line.

Structure and delivery of support

Relatives and REACT supporters were in agreement that REACT would not work as a stand-alone intervention, and that support was crucial. REACT supporters saw their key role as listening to relatives and directing them back to relevant parts of the toolkit at an appropriate pace. Relatives valued support for giving prompts to visit particular modules, elaborating on topics in REACT with additional resources, talking through ways to manage specific scenarios and offering reassurance when appropriate. Although e-mail and other online written communication offered convenience and an opportunity to compose the content, both parties felt that face-to-face (ideally) and/or telephone support was necessary for engagement:

You need at least an initial contact face to face so then you know who you are talking to.

And it did help to – much more so than I think you could do on the phone – it did help to build that rapport and therapeutic relationship with the participant.

REACT supporter role

REACT supporters found adding this role to their existing work a challenge, particularly finding the time to carry out the associated tasks and attending supervision. Some also felt that keeping boundaries between the role and other aspects of their job was not always easy. One supporter struggled with how best to manage negative feedback from a relative about another member of the clinical team. This highlights the importance of allocating REACT supporter roles to staff who have time and appropriate skills, and to ensure that they receive regular support and supervision.

These factors were mapped onto the relevant NPT framework constructs.

Stakeholder reference groups

Stakeholder reference groups met in each participating trust during phase 1 of the IMPART study. The aims were to:

• establish good working relationships with key stakeholders in each trust
• finalise delivery of REACT and IPv1
• explore the views of EIP staff, service users and relatives on factors they thought likely to affect the implementation of REACT in their trust.

Each trust identified an IMPART lead to provide ‘on the ground’ insights into the workings of a particular site and to help researchers to access key data sources. Leads were asked to identify potential SRG members – ideally senior trust board members or EIP service leads, EIP team managers or clinicians, two EIP support workers, two EIP relatives and two EIP service users – and to help set up and co-chair the SRG meeting, held on NHS premises.

All SRG meetings were attended by at least two members of the IMPART research team: a research associate and the North of England or South of England site lead or PPI lead. See Appendix 3 for a detailed report of the methods and results forms including key themes arising from these meetings (see also Appendix 3, Figure 20).

The SRGs were well attended and provided invaluable predictions about factors that would be likely to affect implementation within each trust. However, staff were generally at a clinical and managerial level, with no involvement of senior trust representatives. In some trusts, the IMPART lead chose to hold separate staff and service user/relative meetings owing to fears that meetings would be used to air dissatisfaction with services.

There was a high level of consistency between trusts in terms of the factors considered likely to affect implementation of REACT. These factors included the promotion of its potential to meet the needs of relatives, saving staff time, helping trusts to meet their clinical targets, the flexibility of trusts to adapt REACT to their specific needs, staff confidence in the content and in their role in offering REACT, and the commitment of resources at an organisational level:

We need to make some decisions about the level of support we can give to it and whether or not we can afford to support the instant messaging and the forum, so I guess [we need] to have a clearer idea of what you think the time resource will be.

Stakeholders felt that a key strength of REACT was its availability to relatives without requiring service user consent, as this targeted a currently unmet need.

Initial engagement from staff was anticipated to depend on engendering a sense of ownership of REACT within each team, which would be facilitated by ensuring that staff had easy access to the website and training, and by allocating key people to REACT roles. Ongoing use of the intervention by staff would be facilitated by the ability to monitor relatives’ levels of use, and to change and update the content in response to their feedback:

Will you be using any of the things that get spoken about in the forum to kind of tailor the toolkit? So if there are lots of discussions about sleep or diet will you then use that maybe to add to the toolkit for the future?

Important barriers that stakeholders felt would need to be addressed before staff and relatives would engage were primarily linked to the REACT group (the online forum) and included confidentiality, security and privacy of data (mainly a concern of relatives); the challenge of managing disclosed risk of harm; and the website being used to air negative feedback (mainly a concern of staff).

The latter was particularly evident in one trust and was clearly linked to experience of an earlier online intervention in which some members of staff had been ‘trolled’. Most stakeholders felt that such risks could be managed by restricting access to the forum to invited relatives, and by having clear ground rules and regular moderation by a REACT supporter.

Despite the credibility of the intervention being clearly enhanced by multidisciplinary input to its development, including from service users/carers and health-care professionals, some concerns were expressed about lack of fit between the content of the toolkit and the ethos of the service. In one trust in particular, REACT was perceived as too medicalising by defining diagnostic terms such as schizophrenia. Staff in this trust wanted more content that focused on well-being and recovery than they felt was currently included:

It just seems to me that a lot of the discussion’s been around lifestyle stuff and families wanting to have more information about well-being in general – I just wonder if this is a very medical model and it’s coming from a very kind of, you know, medical perspective, and actually it needs to open up a little bit more to those kind of life.

Challenges were also identified in using REACT to meet the diversity of relatives in terms of language, age, confidence with IT and reading skills. Generally, stakeholders felt that REACT was not suitable for those whose first language was not English, who were older, who were less IT confident or who had a lower reading ability:

[In] my family I’m the only one who speaks English for example, and how would they access this?

Relatives and staff in all trusts were clear that although they would value the information and peer support offered by REACT, particularly if offered early in their journey, it should be offered as part of an integrated package of care and should not replace face-to-face support:

It’s going to tick the ‘information for relatives and carers’ box, but I don’t want it to be instead of actually getting face-to-face contact and support with care co-ordinators and things like that.

Key factors identified by the SRGs as likely to affect implementation of REACT were mapped onto the relevant NPT framework constructs.

Synthesis and generation of study-specific hypotheses

Guided by the online toolkit (www.normalizationprocess.org; accessed 6 December 2019), we began by developing our team-wide understanding of NPT’s four core constructs, then its 16 construct components, and then took the related questions from the interactive toolkit to generate a list of propositions relating to the general processes that NPT predicts are likely to lead to successful implementation of an intervention.

Once the analyses of the systematic review, REACT feasibility trial and SRG data were complete, we used another ‘data day’ to map the key findings from each of these data sets onto the NPT construct components. By using the structure of the NPT toolkit questions, we were then able to synthesise these data sources to generate our propositions specific to the IMPART study (see Appendix 4, Tables 27–30, column 5).

The propositions that resulted from this process under each of the four core NPT constructs are described in the following sections.

Coherence

REACT is more likely to be successfully implemented (i.e. more relatives will be invited to use it) if:

• all staff have easy and independent access to the toolkit in their own time, with clear guidance on what it is, who it is for and what it offers
• staff are given an opportunity to discuss whether or not, and how, to use REACT within their service; decisions from this are visibly endorsed at senior management level
• staff can access training that clearly outlines their roles and responsibilities in delivering REACT
• staff are able to view the content of the REACT toolkit and can see clear benefits for staff and/or relatives, such as –
  • accessibility
  • relevance
  • credibility
  • reassuring
  • non-stigmatising
  • fits with EIP service ethos
  • clear user involvement
  • inclusive
  • safe to use.

Cognitive participation

REACT is more likely to be successfully implemented (i.e. more relatives will be invited to use it) if:

• there is a clear lead (‘champion’) who drives REACT forward within the trust
• staff make explicit links between REACT and existing key trust targets and priorities
• key roles are appropriately allocated
• training is provided for tasks that may require new skills (e.g. forum moderation)
• staff have a sense of ownership and take responsibility for promoting REACT within the service and offering REACT to relatives (and do not see this as the role of the researchers)
• delivery of REACT is included as a regular agenda item at relevant operational meetings within clinical teams.

Collective action

REACT is more likely to be successfully implemented (i.e. more relatives will be invited to use it) if:

• staff carry out their roles and responsibilities as outlined in the ‘how-to’ manual and have the resources to allow them to do this. Key tasks include –
  • IMPART lead creating REACT supporter and clinician accounts
  • all staff inviting relatives (requires access to computer and relatives’ details)
  • REACT supporters regularly moderating the forum (requires regular access to the online forum)
  • REACT supporters providing timely responses to direct messages
  • regularly updating blogs and local resources
• REACT is offered as part of an integrated package of care for relatives, and not a stand-alone intervention
• access is restricted to relatives already being supported within the team
• staff can see and are confident that the forum is being well managed
• staff can see and are confident that direct messages are being responded to
• REACT supporters and IMPART leads are clear and confident about managing risk identified on REACT
• IMPART lead and REACT supporter roles are allocated to people with the time, skills, organisational role and support to carry them out
• REACT is clearly visible within the relevant clinical care pathway trust policy documents
• REACT is customised with accurate trust details
• staff are allocated time for training, supervision and carrying out their tasks specifically related to REACT
• staff have easy access to computers or tablets and IT support to enable their online tasks related to REACT
• staff can work easily between REACT and existing electronic health-care record/IT systems.

Reflexive monitoring

REACT is more likely to be successfully implemented (i.e. more relatives will be invited to use it) if:

• staff can access regular data that show that relatives are using REACT
• feedback from relatives is reviewed and shared as part of an operational meeting and can inform ongoing work
• staff are able to gather direct feedback from relatives they have invited to use the website (if positive, this facilitates implementation)
• staff are able to request or enact improvements to REACT as they see fit.

Discussion

The aim of this chapter was to explain the development of our study-specific model for the IMPART study. We created a list of hypotheses in the form of propositions about factors likely to affect the successful implementation of REACT, drawing on a mid-range theory (NPT), relevant data from a review of existing literature, our previous feasibility study and SRGs in each participating trust.

The process of developing these propositions helped our team to better understand NPT as a theory, and specifically its relevance to the IMPART study, and to operationalise each of the NPT constructs. They have guided our data collection by telling us the kind of data we are likely to need to test each one, and have informed the development of topic guides for staff interviews.

Integrating the three data sets into a NPT framework was not without challenges. NPT’s focus is the work undertaken by staff to deliver the intervention rather than the work undertaken by the user to then engage, although it does indirectly acknowledge the importance of user engagement in the reflexive monitoring of staff. In contrast, a substantial number of our data focused on the engagement of the end user with the DHIs, and so highlighted factors associated with characteristics of the DHIs or of the user. Although it would be possible to adapt NPT to explore the work undertaken by the relatives to engage with REACT, this was beyond the scope of this project. Therefore, data relevant to this engagement is not included in the NPT framework, but is used in Chapter 8 to help interpret the data collected from relatives set out in Chapter 7.

The hypotheses derived from phase 1 are revisited in Chapter 8, and tested in the light of the data collected in phase 2 (see also Appendix 4).