Explanatory synthesis

This chapter integrates the findings from all phases of the study and across all of the data sources, including our observations and insights during the process of the study, to identify the key factors affecting the implementation of REACT.

Our original plan was to develop an explanatory synthesis that best represented these findings, and offer a new framework that could be generalised to other digital technology projects in health service settings. However, there are so many theories and frameworks in the literature, very few of them independently tested, and no guidance as to which is likely to be of most use for a particular purpose. Therefore, rather than develop an additional framework, we examined our findings through the lens of an existing framework: the NASSS framework.74

Normalisation process theory was the theoretical framework that guided our collection and analysis of qualitative data from staff members within the participating NHS trusts. However, this theory did not readily apply to understanding the experiences of relatives, nor did it capture many of the wider contextual factors that were clearly affecting the implementation of REACT. NASSS was recently developed to help predict and evaluate the success of a technology-supported health or social care programme. It was developed from an extensive systematic review of previous technology implementation frameworks and a series of six empirical case studies, each testing a different type of technology-supported programme (e.g. video consultations, pendant alarm systems and care-organising software) in different health-care settings.

The framework, shown in Figure 19, outlines seven key domains that are important to consider to determine the success of implementation:

1. the condition or illness
2. the technology
3. the value proposition
4. the adopter system (staff and patients/relatives)
5. the organisation
6. the wider social context
7. the evolution (interaction and mutual adaptation) of these domains over time.

FIGURE 19 The NASSS framework for considering influences on the adoption, non-adoption, abandonment, spread, scale-up and sustainability of patient-facing health and care technologies. Reproduced from Greenhalgh et al.74 ©Trisha Greenhalgh, Joseph Wherton, Chrysanthi Papoutsi, Jennifer Lynch, Gemma Hughes, Christine A’Court, Susan Hinder, Nick Fahy, Rob Procter, Sara Shaw. Originally published in the Journal of Medical Internet Research (www.jmir.org), 1 November 2017. This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included. (Figure not reproduced here; see the original article.)

The NASSS framework suggests that each of these domains can be characterised by its degree of complexity, and the greater the complexity across the domains, the less successful long-term implementation is likely to be.

The NASSS framework was useful for integrating findings with team learning during the IMPART study. We used it to structure our explanation of key factors affecting the implementation of REACT. As such, we were able to offer an independent evaluation of the applicability of NASSS in a novel setting.

Use of the NASSS framework also enables our findings to be interpreted alongside those of other studies using the same framework. We highlight the issues of complexity in relation to each of the NASSS domains, and identify how this complexity could be better managed. The application of the NASSS framework underpins the presentation of IPv4. Our focus in developing IPv4 was to increase the likelihood of adoption and sustainability of REACT, as the time scale of our study allowed us to capture only this early stage. The content in this chapter is organised around the domains of the NASSS framework.

How successfully was REACT implemented?

We assessed implementation success in terms of how many staff accounts were created, how many relatives were invited and how many relatives’ accounts were activated. However, consistent with the exploratory nature of this study, we had not specified a threshold level for defining implementation as successful. Nevertheless, staff in each trust had implicit ideas about uptake and success of REACT, which affected their behaviour.

Across all six trusts, 355 invitations were sent to relatives, 159 relatives’ accounts were created and 56 relatives completed baseline questionnaires. On average, relatives who created an account visited the website four times and spent 41 minutes on REACT. Given that REACT could potentially have been offered to all relatives supporting someone in the participating teams, these numbers may seem low. On the other hand, many clinicians invited only new relatives who came into the service during the period of the study; these numbers therefore constitute a significant proportion of new relatives.

It is important to note that the aim of our study was not to use researcher activity to maximise recruitment (as we might in a RCT) but to understand what factors affected clinician implementation of REACT. Therefore, we focused on understanding the implementation process of a new DHI in practice and how this differed between trusts.

NASSS domain 1: the nature of the condition

Psychosis and bipolar disorder are both highly complex conditions. The reliability and validity of both terms have been questioned due to the large variation in presentation, outcome and response to treatment in people to whom these labels have been applied.103 The defining features are extreme and experienced largely internally (such as hallucinations, delusions and extreme mood swings). The risks of social isolation, unemployment, self-harm and suicide are high.43,46 Consequently, the challenges faced by relatives supporting someone with psychosis or bipolar disorder are significant and fluctuating. Often, these challenges negatively affect the relative’s own social and mental health.28–34 Of the relatives completing the GHQ-28 in this study, 60% scored about the clinical threshold of 4 out of 5 (items were scored as 0, 0, 1, 1, as recommended for screening) at baseline. Access to support for relatives is highly variable, and despite clinical and government guidelines, many relatives still feel excluded from services.104

On the whole, staff who engaged in the research were very positive about the concept of an online toolkit to support relatives in EIP services, and felt that this would be an appropriate way to meet their needs for information and support. Relatives could also see value in having an online resource that helped them to make sense of the experiences they were dealing with, particularly if it was available at an early stage in this journey.

However, both staff and relatives emphasised the complexity of the issues faced by relatives, and the need for a DHI to complement rather than replace face-to-face support. Ideally, REACT would be part of a package of care, delivered by a clinical member of staff who also worked with the service user and could involve both family members as partners in the ‘triangle of care’.105

Unfortunately, pressures on mental health services meant that staff felt unable to prioritise time spent with relatives. Although most of the relatives we spoke to were very positive about the direct support they had received from EIP teams, staff felt that pressure of workload had led them to prioritise service user outcomes above those of carers. Indeed, some suggested that demonstrating a significant impact of REACT on service user outcomes would give it greater value for staff than any impact on carer outcomes. At an organisational level, there was little evidence of relatives being strategically involved in trust activity. It was challenging to include relatives in the SRGs because none of the trusts had existing frameworks through which relatives could be engaged in service development; some trusts actively resisted inclusion for fear of giving relatives an opportunity to complain about the services they were receiving (see Appendix 3, Table 26).

NASSS domain 2: the technology

REACT was developed over a number of years with extensive user involvement. The content was structured around questions that relatives had identified as important; videos and peer forums facilitated the sharing of stories to reduce stigma, increase hope and exchange practical support ideas; and confidential direct messaging was included to address fears around anonymity. The toolkit was branded with NHS and university logos to enhance credibility and engender trust. Versions for each of the six NHS trusts were adapted with trust logos and specific information to make them locally relevant. REACT was piloted in situ, and staff received online and face-to-face training as part of IPv1.

However, there were several limitations of the technology, which caused staff and relatives to fail to adopt or to disengage from the intervention. Insufficient time and resources led to inadequate internal testing within the trusts. Piloting in situ identified early problems that led to a loss of faith in the intervention and subsequent abandonment by staff. Once the website was live, IT issues continued (such as broken links between pages or to external sites) and caused further loss of engagement.

Considerable time was spent with relatives to ensure that the design of REACT was appropriate. For those who accessed it, this seems to have been time well spent. Relatives appreciated many features, particularly the opportunity to hear the stories of others facing similar challenges. However, staff users had not been involved in designing the interface for referring relatives to REACT and many found it to be unappealing and hard to navigate. It looked technical and some staff felt that they were in the ‘back end’ and feared being able to ‘break’ the website. Staff who persevered and explored the changes made in IPv2 and IPv3 found the website simple to use.

Lack of staff involvement in design also led to a technology that integrated poorly with trust IT systems and with general working practices. For example, accessing REACT required a reasonably secure connection to the web server that some outdated browsers on trust computers could not provide. YouTube videos and some textual content (words such as ‘drugs’) were blocked by trust IT firewalls. The need to log in with a username and password was a major obstacle, with REACT (like all externally provided DHIs) not being directly accessible via the trust login. Despite staff being able to set these individually, remembering multiple logins and persevering when these failed were big barriers.

At an even more basic level, REACT required access to a computer and an internet connection. These were not always available to staff in their place of work and, crucially, were rarely available on mobile devices during home visits to allow them to show REACT to relatives. To preserve the privacy of relatives, staff were also unable to access all REACT components (specifically the forum and direct messaging). This led to anxiety about what might be there and, consequently, a reluctance to invite relatives.

Attempts to improve the usability of the website via an online training manual proved largely unsuccessful, as staff reported it being rarely used (web usage data are not useful here as the page is open access and page counts would include visits by the IMPART team). Face-to-face training was more highly valued, but was attended by only a small proportion of staff at each site. More successful were delegating the role of inviting relatives to a small number of individuals (as in Marsh) and appointing a REACT champion to facilitate training and provide peer support (as suggested in Seashore). Ultimately, though, the lack of engagement with staff in the design was a key factor in non-adoption and abandonment of REACT.

Most relatives in EIP services in all trusts were never sent an invitation to REACT. Those who were experienced similar IT frustrations to the staff. Some never received their invitation, which often went into junk folders (solved by changing the originating e-mail addresses and wording of the e-mails). The same challenges of broken links and forgotten log-in credentials existed.

Initially, relatives were invited to take part in the IMPART research study and complete the pre and post questionnaires as part of the log-in process. This took them to a Liverpool CTRC website that, after completing the measures, redirected them to REACT. On some occasions, this redirect did not work, and some relatives mistakenly thought that the questionnaires were the website. As soon as we became aware of this, the system was changed: new users were taken straight to REACT and the research invitations were sent in a separate e-mail.

A frequently cited reason for relatives (and subsequently staff) disengaging from REACT was the lack of activity on the forums. This was the feature that relatives expressed most interest in and tried most often to access. However, despite some relatives experimenting with posts, the lack of activity was disappointing and led to abandonment. Forums require a critical mass of activity (and therefore people) to be successful. They are very hard to get going and require very active moderators to regularly generate topics of interest and reply quickly to posts, until enough people are active and the moderator can take more of a guiding role. Most REACT supporters saw their role as responding to posts rather than proactively generating activity, and given the small number of relatives invited onto REACT in any one trust, these forums were like empty dancefloors that did not encourage participation.

Lack of activity on the forum created a vicious cycle that reduced staff motivation to offer REACT. Staff were very interested to know what was discussed on the forums and so IPv3 included an e-mailed newsfeed that summarised topics of discussion in the hope of engaging staff and motivating them to refer more relatives to REACT. It seemed to have the opposite effect by highlighting the lack of activity, which staff took to indicate lack of interest from relatives. They therefore stopped referring.

NASSS domain 3: the value proposition

Everyone we spoke to in the IMPART study saw value in having an online self-management toolkit to support relatives.

For senior managers, this was facilitated by the 2014 publication of the AWT standards for mental health106,107 and subsequent audits of the ability of each NHS trust to deliver NICE guideline clinical care for people with psychosis, including carer education and support for relatives. REACT was developed specifically to meet this aim and was presented as such during the initial approach to NHS trusts. Some trusts (e.g. Marsh) were also driven by R&D departments that were rewarded for engagement in NIHR portfolio studies. Consequently, recruiting NHS trusts to the study was easy.

Most frontline clinical staff were aware of trust targets and priorities and of national clinical guidelines, although generally less so for those pertaining to carers rather than service users. For them, REACT’s value was as a high-quality, professional resource that educated both relatives and staff. Some staff felt that this improved the quality of the conversations they were able to have because relatives were more informed. REACT was expected to save staff time in the longer term, and allow services to support relatives of service users who did not want their relative to be directly involved in their care.

Benefits identified by relatives included the ability to access support at a time and location that suited them, and support that was directed specifically at their own needs as carers.

However, perceived value was not sufficient to drive activity. The strongest theme in our data was that staff did not have the time to offer REACT. Although many saw that REACT could save time in the long term, their experience was that in the short term engaging with a new way of working would require additional time and resources. Most respondents in the costs and savings survey taken towards the end of the study did not feel that REACT had reduced the time they would otherwise have used to support relatives. Time is fixed, and staff always have the same amount of time.

Given the evident demands on staff time and the pressure services were under, inability to prioritise delivery of REACT was perhaps not surprising. Teams struggled to provide support to service users and manage risk. Staff morale was low, long-term absences were high and frequent staff turnover was the norm, particularly in Woods and Seashore. Staff talked of the very many targets and ‘priorities’ they were supposed to meet, and how they were consequently forced to prioritise within the priorities. As one IMPART lead in Moor Trust put it, ‘REACT is just not the shark nearest the boat’.

Priority was generally given to targets that were service user focused, financially incentivised, nationally endorsed or ‘home-grown’. REACT was carer focused, had no funding attached to its delivery, was not endorsed by NHS England and had been developed and offered by a research team external to the trust.

Relatives also faced time challenges. These were made easier when REACT’s potential value was made explicit and personal to them when it was being introduced. Particularly, relatives were more likely to sign up to REACT if it was introduced by someone in the service with whom they already had a good relationship, whose opinion they trusted and who spent time explaining what was in REACT and why they might benefit from it.

Timing was also crucial. REACT was more highly valued if offered early in the relative’s contact with services, while they were still trying to make sense of what was happening and before they had spent a lot of time and effort finding out information through other means.

REACT was designed to be an adjunctive resource in a comprehensive service. If NHS trusts struggle to deliver other parts of the service for relatives, then its potential value can become a perceived risk. For relatives, the potential risk was that DHIs such as REACT might replace face-to-face support, which was not what they wanted. For staff too, the risk was that relatives might feel ‘fobbed off’ or discover what other services they should have access to. Their frustration would then be directed back at staff and be difficult to manage.

NASSS domain 4: intended adopters

Adoption, non-adoption and abandonment may all be linked to a user’s view of the condition, the technology and the intervention’s perceived value, as outlined above. However, additional aspects of the intended adopters are also important.

In this context, there were two groups of intended adopters: NHS staff and relatives. Both needed to be willing and able to adopt REACT. We therefore identified IMPART leads in each trust, but left the allocation of key roles including the REACT supporters to the discretion of the service. Our hope was that these would be allocated to the staff perceived as most willing and able, and that early adopters would lead the way for other staff to follow. To some extent, this did happened. However, significant obstacles became apparent.

The main obstacle to staff willingness to refer relatives to REACT was a sense of fear and threat. The most fundamental threat was to staff jobs. DHIs are often presented as tools that will enable staff and make their jobs easier, but this is within a context in which many other jobs are being replaced rather than augmented by digital technology,108 and the exact impact of DHIs on NHS jobs cannot be predicted. The fear that REACT could replace face-to-face clinical therapy was present, and had staff questioning whether or not DHIs could ever be as good as face-to-face support (see particularly Woods and Marsh).

Other potential threats came from the potential to be ‘trolled’ on an online forum, or be held responsible for managing risks disclosed online without adequate training or policies being in place. Trolling was a particular fear in Marsh after experiences with an earlier DHI. Fear of being held responsible for risk events was widespread across all trusts and, in one (Lakes), directly led to staff withdrawing support for REACT. None of the trusts seemed able to reassure staff or produce risk policies that had been adapted for online communication.

Finally, for some staff, REACT also threatened to highlight their service’s existing difficulty in meeting relatives’ needs. It was feared that REACT would open up a line of communication with relatives who would be better educated about what services they should be receiving, and consequently request access to services the trust was unable to offer. Staff felt that it would have been difficult to manage these expectations.

Some staff did not adopt REACT because they did not see it as legitimately fitting their role. Either it was perceived as belonging to a different professional group, such as ‘led by psychiatry’ or ‘from psychology’, or it was seen as the role of the research team to deliver REACT. Staff were more familiar with research designs such as RCTs where they would identify potential participants but in which the research team would recruit the participant and deliver the intervention.

Staff had to be able as well as willing, and specific DHI-related skills were important. General confidence in using DHIs was surprisingly low, and fears of ‘breaking the site’ or getting it wrong were not uncommon. Navigating the website and remembering log-in details were challenging, made more difficult by the limitations of the technology design outlined in NASSS domain 2: the technology. Despite 281 staff accounts being created, only 57 staff members ever sent a relative’s invitation.

If relatives who were sent an invitation or made aware of REACT were willing, their ability to navigate the process was also a significant barrier, again mainly owing to the technology design outlined in NASSS domain 2: the technology. Some relatives tried to find REACT through a search engine rather than by following the link in their invitation e-mail. This led them to the REACT trial website, which was running at the same time; when they tried to log in, their details were rejected, causing frustration and abandonment.

NASSS domain 5: the organisation

REACT was designed to help NHS trusts to deliver NICE guideline care by offering carer education and support alongside other recommended interventions, such as structured, face-to-face family interventions; carer support groups; and joint case management. The website could accommodate trust-specific information, such as availability of website moderators, or other aspects of the care and resources directory. Staff feedback in individual interviews and workshops led to improvements to the interface and a planning document to support trusts in thinking through how REACT could best fit their existing service structure and care pathways.

Despite this, there were a number of ways in which REACT did not ‘fit’ with services, impeding implementation. Failure of fit was evident at the level of organisational culture and the specific clinical pathways into which REACT was offered.

Fit with organisational culture

Adopting a DHI into EIP teams required a huge shift in working culture. These teams were historically characterised by people talking to each other and recording activities using pen and paper. Direct human contact was understood to be the main agent of change. An online intervention was not something staff were naturally exposed to or generally seeking. Basic access to IT facilities and staff training was extremely limited, and experience of other digital health systems, including electronic health records, had been generally frustrating.

During our data collection, it was clear neither to us nor to staff who was responsible within trust organisations for the strategic direction of service development and who could facilitate the shift needed for staff to embrace DHIs such as REACT. Many were not sure who needed to give the ‘go ahead’ for staff to use REACT, and the distinction between clinical and operational managers seemed to add to the confusion. Some trusts (e.g. Marsh) had a ‘transformation manager’, but it was not clear how they would support the use of interventions such as REACT.

REACT came into trusts via R&D teams that were very keen to support research in their trust. Although the clinical teams agreed to this happening, often the decision was made by a key senior individual (who often then became the IMPART lead) without much evidence of team consultation. In one trust (Lakes), this person then left the team during the study period.

Several consequences stemmed from this lack of strategic engagement. First was that teams were sometimes being asked to adopt several new approaches in parallel, and change in many directions at once. For example, staff in Seashore were being asked to integrate a new DHI to support service user recovery at the same time as learning about REACT, which caused some confusion. Second, because the intervention had no clear clinical ‘home’ and was not adopted by any specific professional roles, there was no infrastructure or allocated resources to support ongoing sustainability or spread. REACT supporters were not given regular supervision to support their work (> 50% of REACT supporters responding to the costs and savings survey had received < 1 hour of supervision for their REACT role over the entire study period) and, consequently, there were no incentives for them to carry this out. Over time, REACT supporters disengaged from the intervention, leaving the forums often unmoderated. This disengagement exacerbated the fears of clinical staff regarding risk and trolling, which in turn reduced the likelihood of staff to refer relatives to REACT.

In all trusts, there appeared to be a lack of organisational capacity to strategically change practice: to reflect, adapt and reorganise. We noted that teams were constantly fighting to meet immediate service user priorities with low staffing and limited resources. Reported caseloads were higher than the EIP model was designed to accommodate. There was no evidence of any positive impact from the additional EIP funding or the government’s commitment to ‘parity of esteem’ for mental and physical health (although we did not collect comparative data for any physical care services).

Fit with the clinical pathway

In general, EIP teams offered support to relatives in the relative’s own home. Most staff did not have regular access to mobile technology on which they could show REACT to relatives in these face-to-face consultations. REACT was literally out of sight and, therefore, very often out of mind. Even when staff brought REACT to mind, social barriers impeded them from introducing it. Although most relatives had personal computers or tablets, staff did not feel that it was appropriate to ask the relative to log in to their own personal device to be shown the website, particularly as these were often in the bedroom or other private area of the house.

We tried to address this issue by providing (in IPv2) attractively designed REACT NHS booklets that explained what REACT offered and how to access it, and later (IPv3) printable PDF summaries of the REACT modules. These were generally welcomed by staff, and highlighted the complementary benefits of using digital technology alongside other forms of communication.

Staff who did refer relatives to REACT had no subsequent knowledge about whether or not the relative had accepted their invitation and used REACT and, if so, what their impressions of it were. This lack of feedback was demotivating, and led to disengagement among staff. As noted above, our attempt to address this via the newsfeed in IPv3 seemed to have the opposite effect as it highlighted the lack of activity.

Finally, at the time of the study there was no formal way for staff to record REACT-related activity in their electronic note systems, other than as a note in the service user’s electronic record. In all trusts, any staff time spent supporting relatives was recorded in the service user records, as carers do not have an independent record. Consequently, staff caseloads did not accurately reflect the number of people a staff member was working with. As the pressure on systems continues to grow, the cliché that ‘what gets measured gets done’ is likely to become increasingly true.

NASSS domain 6: the wider system

REACT fitted very well into several key government agendas. It had the potential to facilitate a shift towards a paperless NHS by 2020 as set out by NHS Digital;12 it supported the delivery of NICE-recommended care, to increase parity of esteem between mental and physical health;106 and it also had the potential to facilitate meeting the needs of carers as set out in chapter 23 of the Care Act 2014.109

However, despite this, key factors within the wider system worked against the implementation of REACT.

Despite the commitment of large amounts of money to meet the NHS’s commitment to ‘harness the information revolution’,12,110 our findings suggest that access to up-to-date mobile hardware and software was very limited for frontline NHS staff. Confidence in using digital technology was low and there was no evidence of training or support available within these EIP teams. Anxiety about responsibility for risk was very evident and was situated within a wider context of societal anxiety about data security111 and a blame culture in the NHS.112

Despite the inclusion of carer targets within the AWT framework, it was difficult for services to prioritise activity to meet these. Our health system reflects our societal system in which health is generally conceptualised within an individualistic framework, in which the focus is on the individual rather than the social structure in which they are situated.113 Relatives in the UK provide an estimated £1.24B per year of support to the NHS in unpaid caring roles,27 and staff understand the logic of supporting them in terms of longer-term outcomes. However, failure to prioritise carer support seems to reflect the lack of organisational capacity to do much more than short-term risk management. The services taking part in our study were extremely stretched. Although mental health is a government priority, and funding has been targeted towards EIP services more than at other parts of mental health services,106 this is within a context in which annual rates of increase in the overall health budget to meet growing demand have been declining over the past decade.114

NASSS domain 7: evolution over time

One factor affecting staff non-adoption and abandonment of REACT was that it was perceived as part of their trust’s research agenda rather than as a long-term clinical strategy. However, several staff asked whether or not REACT would continue to be available to them at the end of the project. REACT was wholly funded as NIHR research and, therefore, we were unable to guarantee that REACT would continue to be available and supported. Understandably, this limited the effort and behavioural adaptation that staff were willing to invest in something that might disappear. Some staff questioned whether or not, in this situation, REACT should be offered at all.

There is currently no national infrastructure for tested products funded by public money to be adopted and delivered nationally at the end of the research phase. Survival depends on the developers (who are often clinical academics and not entrepreneurs) finding collaborative partnerships with digital technology partners, private investors or individual commissioners at a trust level, and setting up commercial contracts. REACT was being tested in a large RCT in parallel to this study and, without the data from that research, it was not possible to establish a longer-term delivery model or establish funding for it. We are not alone in facing this challenge. A recent systematic review of meta-analyses of web-based interventions developed and tested in RCTs showed that in only 21.3% of cases (57/268) did a functional website continue beyond the trial.115

It must be noted, however, that as well as discouraging staff engagement with REACT, the research process also triggered activity. Contact with staff to arrange individual interviews and SRGs was associated with an increase in relatives’ invitations (see Figures 8–13). Similarly, there was some suggestion from the web use data that relatives who signed up to take part in the research study also visited the website more often, although this may just reflect a general tendency to comply with requests.

Conclusion: implementation plan version 4

The NASSS framework hypothesises that the degree of complexity within each of its seven domains will determine whether or not a DHI is successfully implemented. Each domain can be assessed as simple (few components and predictable, e.g. making a sandwich), complicated (lots of components, largely predictable, e.g. building a rocket) or complex (dynamic, multiple interacting elements and unpredictable, e.g. raising a child).116 However, the domains are interdependent and so it is not possible to identify any one specific factor that underpins implementation success.

In the present study, challenges were identified in all of the seven NASSS domains. The level of complexity in each domain was high. Although this gave clear direction as to what needed to change, it was also somewhat disheartening. Although we might be able to improve the value of REACT and the reliability and interoperability of the technology, build greater capacity to use it in staff and relative adopters, and may even be able to offer support for organisational change at a trust level, we are much less able to reduce the complexity of psychosis (the condition), the wider context of NHS funding or the lack of infrastructure to support long-term funding of DHIs (emergence over time). Despite the apparent culture of in-house development in large technology companies of ‘fail early, fail often’, because of the way in which competitive funding operates in the public health sector, there is no reward for sharing learning about how to overcome barriers, and only DHIs presented as successful are likely to attract further funding.

We have highlighted the main factors that our findings suggested were important in relation to each of these domains, and used them to develop IPv4, set out in Table 24. The strategies map directly onto the factors raised in each of the seven domains. Finally, we have drawn out recommendations that can be used to inform the implementation of other DHIs within similar community health teams.

Suggestions for others implementing digital health interventions based on the findings from the IMPART study

The following recommendations are based on the key findings of the IMPART study. They are aimed at people designing and delivering DHIs within the health-care sector and, for this reason, they complement the broader recommendations of Greenhalgh et al.74

• All stakeholders, including referrers and end users, should be closely involved in the development, testing and implementation of DHIs at every step of the design process.
• Allow sufficient time and resources for iterative co-design, internal testing and piloting (and then some more).
• Identify the potential value of the technology to each stakeholder group within the health-care setting and provide evidence (including data, endorsements and testimonials) that directly targets these values.
• Understand exactly when, where and how it fits into the care pathway and conduct a risk assessment of all the steps along the pathway to see where implementation could be impeded.
• Ensure that all stakeholders are actively involved in the service decision to adopt the DHIs, and that they have the opportunity to have their fears and concerns heard and addressed prior to adoption.
• Understand how change happens within an organisation and ensure that the key agents of change are engaged in the implementation process.
• Identify champions within the workforce and user groups who will take a lead role in ensuring ongoing sustainability of the intervention.
• Manage expectations about what can realistically be achieved with the technology.
• Be transparent about the short-term and long-term costs and savings of delivering the technology. Be careful to include the costs incurred in changing existing practice and training staff in new ways of working.
• Understand the hardware and software that the organisation is using and design the technology appropriately.
• Establish a clear communication channel for ongoing stakeholder feedback, and a mechanism to ensure that this can be addressed in a timely manner.
• Develop a comprehensive training programme for all staff and service users involved in the DHI. This should include a balance of the formal (built into mandatory organisational training programmes) and informal (peer-to-peer support and demonstrations), and should be offered on a rolling basis to ensure that skills are regularly updated and new staff can easily access training.
• Training should include general IT skills to increase confidence, as well as skills specific to the DHI, and clear guidance on management of risk and responsibility, and should be offered through a range of media (online, face to face, paper) to support different styles of learning.
• Ensure that the technology is rigorously tested internally and in situ, while managing expectations to ensure that confidence in the technology is not lost during the process.
• Engage early with the evolving infrastructures and support systems surrounding digital interventions, including the Academic Health Science Networks, NHS Innovation Accelerator, the national accelerator programme, the NHS Technology Adoption Centre117 and others, to establish long-term delivery models as soon as possible.
• Build in efficient and agile evaluation of the reach and impact of the technology, complying with General Data Protection Regulation101 requirements for the collection of personal data.
• DHIs are often designed to add value to multicomponent packages of care. It is important to establish that those aspects of care are reliably delivered before assessing the impact of the DHIs.

These recommendations should be considered by those attempting to design or deliver DHIs within a health-care system. However, for DHIs to have significant impact on the UK health-care system, a shift in policy and practice is required at a national level. Our recommendation is that DHIs that are developed with public funding should be made freely available to the NHS. Those that are effective should be adopted and made available via a national and centrally run organisation (such as NHS Digital), rather than commissioned at a local level. Equally rigorous standards should be set to evaluate the effectiveness of commercially developed products, and careful contracting should prevent the risk of price inflation once the health-care system has become reliant on the DHIs.