Phase 2: methods

This chapter details the methods used for phase 2 of the IMPART study. The primary aims of phase 2 were to:

• test our hypotheses about which key factors would influence the implementation of REACT across our six participating NHS trusts
• develop an implementation plan.

An additional aim was to assess the effectiveness of REACT. We follow StaRI71 where appropriate.

Design

As described in Chapter 2, the design was a theory-driven multiple case study design72 using a mixed-methods approach integrating quantitative assessments of outcome (delivery, use and impact of REACT) and qualitative assessments of mechanisms including observation, document analysis and in-depth interviews with staff and relatives.

Context

The study took place across six NHS trusts in England. This context is described in detail in Chapter 1.

Sites

Preserving the anonymity of participating trusts and teams was of particular importance in this study owing to the sensitive nature of the data collected from carers and staff. A multilayered taxonomy of birds and habitats was used to identify trusts and teams within each trust.

Trusts were selected to be geographically and ethnically diverse to maximise the generalisability of findings. A summary is given in Table 3, drawing on the Royal College of Psychiatrists College Centre for Quality Improvement (CCQI) audit in 2014;92 specific figures are not provided so that trusts are not easily identifiable.

Participants

Participants were all staff in EIP teams and relatives or close friends of someone with bipolar disorder and/or psychosis, referred by staff to use the REACT toolkit, and who then consented to take part in the study (subsequently referred to as research relatives). Relatives who did not wish to take part in the research were still able to access REACT and their data were used only in summarising levels of clinical activity (subsequently referred to as clinical relatives).

Staff were initially informed about REACT and the IMPART study through the IMPART leads. Staff and relatives were invited to SRGs and to other group workshops by the IMPART leads, with support from the research team. Individual staff were approached directly by the research team and invited to take part in individual interviews. A detailed description of sampling and procedure for individual and group interviews is given in Process evaluation.

Relatives were invited to use REACT via an invitation e-mail from a staff member, with instructions on how to set up an account. Relatives who visited the website were also invited by e-mail to take part in the collection of research data, and had the choice of completing questionnaires, taking part in an interview (see Chapter 7) or both. Relatives could use the REACT intervention without participating in the research, although all REACT users had to agree to terms and conditions that permitted collection and analysis of basic demographic and website use information. Relatives who agreed to participate in the research provided informed consent online before being asked to complete any research measures.

Description of the intervention

The REACT intervention is described in detail in Chapter 1.

A separate instance of the REACT toolkit was set up for each of the six NHS trusts. This allowed each trust to manage its own website, restricting access to invited relatives. Each trust was clinically responsible for its website, including moderating the forum and responding to direct messages.

Description of implementation plan version 1

The first version of the implementation plan, IPv1, was informed by the hypotheses described in Chapter 3, and designed to be deliverable at minimal cost. It had the following components, each promoting NPT core constructs.

Personalisation of the intervention to each trust

This was intended to engender a sense of ownership and legitimisation. Each trust was able to customise some elements of the website content including adding a logo, times that the forum/direct messaging services were actively moderated and by whom, emergency contacts, and photos and biographies of the trust’s REACT supporters and IMPART leads.

Initial familiarisation with the intervention

This was aimed at building coherence and cognitive participation.

Pilot REACT

All six trusts received pilot versions of REACT on 19 September 2016. During this pilot phase, the research team responded to feedback from the trusts.

Training

An online manual of instructions for delivery and use of REACT was provided to all staff in the pilot phase, and was updated throughout the study in response to feedback. Each trust had at least one face-to-face training session, offered as soon as possible after the website went live. Some had more sessions to accommodate separately located EIP teams. The training provided an overview of the importance of carer information and support and NICE guidelines, an outline of key components of the REACT website, aims of the IMPART study and key roles, and how to use REACT.

Establishing local champions

This promoted cognitive participation. An IMPART lead was identified in each trust to work with the research team to assist with the research process, and within the trust to implement REACT.

Each IMPART lead held a senior role in the EIP and had expressed interest in the study, but roles and selection varied between leads. For Moor and Woods Trusts, the IMPART leads were co-applicants of the project and senior clinicians in their trusts. In Ocean, the first IMPART lead had been involved in the development of REACT. In Seashore and Marsh, senior clinicians were recruited; in Lakes, a senior psychiatrist with an interest in research took on the role.

The IMPART leads were encouraged to customise their trust’s REACT website with support from the research IT team. They also helped set up a SRG of staff and service users or relatives in each trust as described in Chapter 3. These groups met periodically throughout the study, initially to support the research process, but quickly evolved to become an important data source. Finally, IMPART leads were often instrumental in allocating REACT supporter roles within the EIP service.

Establishing workflows

This promoted coherence, activation and reflexive monitoring. Staff needed accounts in order to visit the REACT toolkit, invite relatives and monitor whether or not their invitation had been accepted. An account was created by the research team for each IMPART lead who then created REACT supporter accounts for the people moderating the website and clinician accounts for staff who could refer relatives to the website. Clinicians then created accounts for relatives. This ensured that control over access to REACT remained with the trust staff. Table 4 shows each of the key roles related to REACT and the different levels of website access for each role. Figure 6 shows the pathway through the REACT website.

FIGURE 6 Pathway through the REACT intervention. CTRC, clinical trials research centre. (Figure not reproduced here; see the original article.)

Go live

REACT went live for all six trusts on 21 November 2016. Trusts were informed by an e-mail sent to each IMPART lead explaining that feedback on the pilot had been addressed and REACT was live in their trust. Instructions on how to deliver and use REACT were available via the online manual, and face-to-face training was available via the research team. This was followed up by telephone to arrange training. Figure 6 shows the pathway of how relatives were invited to use REACT.

Developing and delivering the implementation plans

All trusts received IPv1 at the start of wave 1. IPv2 was developed with stakeholders during waves 1 and 2 and delivered to the two trusts in wave 2 (Ocean and Seashore). IPv3 was further developed from additional learning in waves 2 and 3 and delivered to trusts in wave 3 (Lakes and Marsh). The content of these implementation plans is described in detail in Chapter 6.

The final implementation plan (IPv4) was also informed by data from relatives’ interviews (see Chapter 7) and by the development of our explanatory synthesis (see Chapter 8). Figure 7 summarises the iterative ‘You said . . . we did’ action cycles that underpinned this process.

FIGURE 7 Iterative cycles used to develop REACT implementation plans. (Figure not reproduced here; see the original article.)

Each action cycle began with a workshop to which EIP staff and relatives were invited to share their experiences of using or not using REACT, and to explore the facilitators of and barriers to use. The workshops were co-facilitated by a member of the research team, the IMPART lead and a carer researcher, and resulted in an agreed list of changes that would better facilitate the implementation of REACT. Staff were encouraged to talk openly about all the challenges they faced, but the workshop facilitators attempted to pull out those challenges that there was some possibility of addressing as part of the implementation plan. Staff were encouraged to expand on issues in individual interviews such as high workloads, lack of IT training and staff turnover, but the workshops aimed to be solution focused to identify practical changes that could be made.

These changes were shared by e-mail and phone with workshop attendees and those unable to join, and further suggestions were invited. Any changes requiring adaptations to the REACT interface were reviewed by the research team with the IT developer to assess feasibility and time scale. Agreed changes to the interface and to the implementation plan were tested and refined with workshop attendees and within the research team before being implemented. Any changes that could not be made were clearly identified alongside reasons why this was not possible. Observations and feedback on each implementation plan were collected to inform subsequent iterations.

This process required fast, agile working, identifying and responding to needs within very short time scales. For this reason, data were analysed at two different levels, described in detail in Analysis.

Evaluation

Implementation outcomes were measured using primarily quantitative measures from clinicians and relatives. Intervention (REACT) outcomes were measured using relatives’ questionnaires covering five areas:

1. distress, using the General Health Questionnaire-28 (GHQ-28)93
2. well-being, using the Carer Well-being and Support Scale (CWS)94
3. quality of life, using the EuroQol-5 Dimensions, five-level version (EQ-5D-5L)95
4. eHealth literacy, using the eHealth Literacy Scale (eHEALS)96
5. relatives’ caring role.

Process evaluation to identify the factors affecting implementation of REACT was understood using primarily staff interviews and workshops, SRGs and some document analysis and meeting observations. We also evaluated the resources required to deliver REACT using an online questionnaire. Study objective and data collection are described in Table 5.

Implementation outcomes

Clinicians’ activity was monitored throughout the study in each trust to assess their engagement with the REACT toolkit, as quantified by the number of accounts they created and the number of invitations they sent to relatives.

We collected basic demographic and web use information from all relatives on the REACT website (research and clinical relatives) to evaluate which users the intervention reached and which modules were most and least visited. The demographic questionnaire [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] completed at the time of creating an account collected participants’ age, gender, ethnicity, highest education, current employment status and internet access. Website use data were collected during the study period, including the total number of logins to the toolkit, the total number of visits to each module and to the forum, and the total time spent on the website.

Intervention outcomes

Those who agreed to complete the outcome measures (research relatives) provided informed consent online before completing the questionnaires outlined above. These measures were chosen because they have good psychometric properties and were acceptable and sensitive to change in the REACT feasibility trial.53 Questionnaires were offered at baseline and again after 12 and 24 weeks to ensure that we captured short- and longer-term impacts. Participants received three automated reminder e-mails to complete the questionnaires at each follow-up point. Owing to limited resources, no further efforts were made to follow up non-responders. All data from relatives were collected online, anonymised and downloaded to a database held at the Liverpool CTRC.

Measures

General Health Questionnaire-28

The GHQ-28 assesses the general health of participants during the previous few weeks across 28 items, each assessed on a 4-point Likert scale ranging from ‘not at all’ (0) to ‘much more than usual’ (3). Potential total scores range from 0 to 84, with higher scores indicating worse health outcomes. There are four subscales of seven items each, covering somatic symptoms (items 1–7), anxiety/insomnia (items 8–14), social dysfunction (items 15–21) and severe depression (items 22–28). The tool is also suitable for screening purposes using an alternative binary scoring method (with the two lowest response options scoring 0 and the two highest options scoring 1). Using the latter method, any total score exceeding the threshold value of 4 would classify the respondent as ‘caseness’, indicating that further medical attention was required.97

Carer Well-being and Support Scale

This tool evaluates relatives’ well-being in the previous 4 weeks and general satisfaction with the support they receive. It has two separate subscales: well-being (32 items) and support (17 items). Well-being is assessed on a 5-point Likert scale from 0 (concerned ‘a lot’) to 4 (‘not at all’ concerned). Support is measured on a 4-point scale from 0 (‘very dissatisfied’ with support) to 3 (‘very satisfied’ with support). The subscales are obtained by adding the item scores, which give a range 0–128 for the well-being subscale and 0–51 for the support subscale. Higher scores indicate better outcomes for carers with regard to their well-being and satisfaction with support.94

EuroQol-5 Dimensions, five-level version

The EQ-5D-5L is a measure of health-related quality of life and includes a descriptive system and a visual analogue scale (VAS) (EQ-5D-5L VAS). The descriptive system evaluates five dimensions (mobility, self-care, usual activities, pain/discomfort and anxiety/depression) on a 5-point Likert scale from 1 (‘no problems’) to 5 (‘extreme problems’). Participants are asked to indicate their health states on the day of the assessment by selecting the most appropriate statement for each dimension. Answers across the five dimensions can be combined into a five-digit profile number. The UK index score value sets range from –0.594 (where all five dimensions score 5: ‘extreme problems’) to 1 (where all dimensions score 1: ‘no problems’); a higher index score indicates a better outcome for respondents. The EQ-5D-5L VAS asks participants to rate their health on the day of the assessment on a 20-cm vertical VAS, scored from 0 (‘the worst health you can imagine’) to 100 (‘the best health you can imagine’). The selected scores can be used as a continuous measure of health.95

eHealth Literacy Scale

The eHEALS evaluates participants’ eHealth literacy, including knowledge, comfort and perceived skills in finding, evaluating and applying electronic health information to health problems. The tool operates with eight items scored on a 5-point Likert scale ranging from 1 (‘strongly disagree’) to 5 (‘strongly agree’). Responses are summed for a total score from 8 to 40; higher scores indicate more positive evaluations of the participants’ own health literacy skills.96

Process evaluation

Factors affecting implementation outcomes were explored through individual staff interviews, SRG workshops, document analysis and meeting observations, and analysed together within the NPT framework, then were used to inform iterative versions of the implementation plan. Interviews with relatives explored their perspective of the implementation of REACT, but were analysed separately using an open thematic coding. Each step is described in the following sections.

Staff interviews

Sampling strategy

A purposive sampling strategy was used to identify key stakeholders, including IMPART leads, REACT supporters, team managers and frontline clinical staff generally referred to as care co-ordinators. We also interviewed people with relevant strategic roles who were likely to play a part in implementing the toolkit in each trust, such as R&D leads and transformation managers.

Quantitative data (e.g. the number of relatives each staff member invited to REACT) and qualitative data from SRGs, workshops and staff interviews were used to identify staff members who were perceived as instrumental to the delivery of REACT, as well as those who were less inclined to promote the toolkit. Opportunity sampling was also used to recruit staff members when delivering training or presentations at trusts.

Development of topic guide for individual staff interviews

The staff interview topic guide [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] was designed to identify factors affecting implementation of REACT in routine EIP service practice and to test specific mechanisms that were hypothesised to be important. The schedule was broadly structured around NPT’s theoretical framework and the study-specific hypotheses generated in phase 1, but also aimed to elicit important contextual information. The guide was further developed in waves 2 and 3 to test ideas that emerged in interviews and case studies, and to assess the impact of new components in subsequent implementation plans. A separate pro forma was developed for IMPART leads [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] with contextual information about the EIP and trusts.

Process of staff interviews

We conducted semistructured, audio-recorded interviews with staff members face to face or by telephone at a time and place of their choosing. Although interviews were focused on testing our implementation hypotheses, the topic guide was used flexibly to allow new ideas and contextual information to emerge.

Before the interview, staff members were given a participant information sheet (PIS) [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] and invited to ask questions before giving written consent [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)]. The PIS made clear that interviews would be recorded and advised that written notes could be taken if preferred. Participants were informed that their data would be anonymised and that anonymised quotations might be used in resulting publications and presentations. Transcripts were cleaned and anonymised prior to analysis. Further data security details are found in Data storage and management.

Meeting observations

The observation pro forma [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] was derived from Spradley and McCurdey’s98 nine dimensions of descriptive observation and adapted for the IMPART study to test propositions in each context. We aimed to observe naturally occurring meetings suggested by the IMPART lead and SRGs as a potential source of data about the context in which REACT was being implemented. Workshops and presentations were also recorded on observation pro formas.

Document analysis

We aimed to sample EIP, trust and national policy documents that might provide information about how relatives were supported in the trust, and specifically if and how REACT fitted into this strategy. All documents were imported into NVivo version 11 (QSR International, Warrington, UK) and coded as one of two codes: ‘What is the trust offering for relatives?’ and ‘Document’s reference to REACT’. The document analysis supported the interview data by providing contextual information for the implementation of REACT. The documents were also used to identify and compare similarities and differences between trusts.

Reflective diaries

Reflective diaries were kept by the lead researchers throughout the study. They were used to develop the researchers’ interpretation of the data as they were being collected, including contextual non-interview data that were informative to the overall narrative understanding of the process of implementing REACT (e.g. summaries of relevant e-mail correspondence or telephone calls). The diaries also recorded timing of key contextual events to understand how these affected implementation outcomes. Most importantly, these diaries provided a real-time record of the impressions and insights of the researchers on the ground to ensure that this could most effectively inform the final analyses. This was particularly crucial given the change of research staff, meaning that researchers who were closest to the data in wave 1 were no longer working on the project at the time of the final analyses.

Stakeholder reference groups

Stakeholder reference groups took place in all six trusts at key points throughout the study (see Report Supplementary Material 1 for dates). The first groups are described in detail in Chapter 3 and Appendix 3 and were used to inform the development of our study hypotheses. Subsequent SRGs were primarily used to develop iterations of the implementation plan, providing feedback on what was already provided, identifying ongoing challenges and co-developing subsequent versions. The attendees were largely determined by the IMPART leads in collaboration with the research team.

Relatives’ interviews

Sampling

Relatives who consented to complete online questionnaires were also invited to take part in an interview about their experience of being offered REACT. Partway through the study, we applied for an ethics amendment to separate the process of inviting relatives into the research study and inviting them into REACT. This change was in response to feedback from relatives who were confused about whether or not the research questionnaires formed part of the REACT intervention. The baseline research questionnaires were removed from the initial log-in process, and relatives were given the option of completing these later and/or being contacted about taking part in an individual interview or declining to take part in any form of research. The amendment also approved recruitment (via EIP staff) of relatives who had been invited to use REACT but who had not proceeded to access the website and create an account. Our aim was to sample across all six trusts.

Procedure

This part of the study was led by a trainee clinical psychologist (Johanna Barraclough), supervised by Fiona Lobban, and closely supported by the carer researchers and lead researchers at each site, with additional input at the analytic stage from the wider PMG.

Relatives who consented were contacted by a member of the research team by e-mail or telephone, sent a copy of the PIS and consent form [both available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)] and offered the opportunity to ask questions. For relatives who agreed to take part, an interview was scheduled at a time and location convenient to them.

Data collection

Interviews took place between May 2017 and May 2018 by telephone or face to face in NHS clinical locations, and lasted 40–70 minutes. Interviews were semistructured in accordance with the interview topic guide [available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020)], with flexibility to allow issues raised by participants to be fully explored. Topics explored with relatives included their caring role, the process by which they were first introduced to REACT, their initial impressions, their use of REACT, REACT’s pros and cons, any impact that REACT had on them or their family, how REACT compared to other forms of support they had accessed or been offered, and support for REACT from the EIP team. All interviews were audio-recorded and transcribed.

Resources survey

At the end of the study, clinicians were asked to complete an online cost and savings survey to estimate the time and resources used and/or freed up by using REACT in their services. This is available at www.journalslibrary.nihr.ac.uk/programmes/hsdr/140416/#/documentation (accessed February 2020).

Sample size

Our main purpose was to understand the process of evaluation, building a detailed picture in each trust by talking to relevant staff in EIP teams and in the wider organisation. We attempted to sample relatives, including those who had used REACT and those who had not, to interview across all participating trusts. The sample of relatives completing the quantitative measures was determined by the level of implementation.

Analysis

Quantitative data

Descriptive statistics for variables measured on an interval scale were mean, standard deviation (SD), median, range and interquartile range (IQR); for categorical (including ordinal) variables, they were frequency (n) and percentage. Pearson’s correlation coefficient was used to quantify linear associations between interval variables, with Spearman’s rank correlation coefficient being used if the association was potentially non-linear. Independent sample t-tests (using the Satterthwaite approximation for the degrees of freedom where equality of variances could not be assumed) and Pearson’s chi-squared tests were used to explore differences between clinical and research participants. Paired t-tests were used to investigate the mean change from baseline to 12 weeks and to 24 weeks in the research participants’ self-reported scores; effect sizes were estimated by dividing the mean change by the baseline SD. Statistical significance required a two-sided p-value of < 0.05, and 95% confidence intervals (CIs) were presented. All data analyses were performed using IBM SPSS Statistics version 23 (IBM Corporation, Armonk, NY, USA).

Qualitative data: staff

Qualitative data sets collected from trust staff, meeting observations and document analyses were analysed using framework analysis85 and NVivo software. Data were analysed on two levels.

Level 1 analysis

This was a dynamic and agile sense-making process that provided the basis for preliminary explanation building that informed the development of each implementation plan. The analysis drew on themes that were clearly emerging from the data prior to detailed coding into the NPT framework. Themes were identified by the lead researchers at each site, and discussed during fortnightly analysis meetings, and at the monthly PMG meetings. The key factors were then presented and discussed in a workshop setting with key stakeholders from the subsequent wave for further elaboration and testing as part of developing the subsequent implementation plan.

Level 2 analysis

In parallel to the level 1 analysis, all data were uploaded into NVivo and coded line by line. A framework analysis was used to synthesis data from different sources and accommodate multiple researchers’ contributions. The initial framework was developed from NPT and findings from phase 1, then further developed during the study as codes were refined and additional codes were added. Changes to the framework were agreed at fortnightly coding meetings, attended by the lead researchers for the North of England and South of England sites and the chief investigator.

Researchers analysing the data at each site had access to NVivo files, which they used to store and analyse data. At regular intervals, these files were merged, collating the data from the two lead researchers. Previous files were archived and researchers each worked on an individual copy of the most recently merged document. This process was repeated throughout data collection and analysis.

At the end of each wave of phase 2, individuals from the PMG attended one of two ‘data day’ events to analyse the qualitative data from one of the trusts in that wave. Six trust data days were held, one for each trust. To ensure that the analysis had a broad range of perspectives, each was attended by the North of England and South of England lead researchers, the chief investigator, at least one carer researcher and at least one other clinical academic expert. Anonymised summaries of the data were distributed for familiarisation prior to the event. The day then involved the PMG members discussing the themes emerging in each part of the coding framework, the differences between embedded units in the trust and the general narrative emerging from the trust summaries. Areas requiring further exploration were also identified and necessary changes were made to the interview topic guides.

Qualitative data: relatives

Analysis of relatives’ interview data was led by Johanna Barraclough and followed the procedure proposed by Braun and Clarke.99 This inductive thematic coding method allows themes to emerge from the data rather than relying on a pre-existing framework, but also offers a detailed description of the process that was easy for our carer researchers, who had no formal research training, to become involved in, ensuring multiple perspectives.

The first phase of analysis consisted of thorough immersion in the data by reading the entire data set of interviews several times and recording notes about ideas and meanings in a reflective journal. Ideas about interesting segments of the data were organised into initial codes, which represented areas of meaning. The entire data set of interviews was worked through methodically, with all potential codes marked in the margin of each transcript. A pictorial map of codes was created using sticky notes, enabling visual identification of patterns of meaning while retaining sight of instances of departure. To ensure that the process was valid and rigorous, two members of the research team with experience as carers of a close relative with psychosis or bipolar disorder independently coded six transcripts, each with the results discussed with the lead researcher (Johanna Barraclough). A high level of consistency provided reassurance that the process was appropriate, and discussions about the implications of the codes identified served as a useful foundation for building themes, with the lived experience perspective of the carers providing insight.

The codes were then compiled into a Microsoft Word (Microsoft Corporation, Redmond, WA, USA) document together with supporting quotations from interviews.

The next phase involved taking a broader perspective on the data and grouping the codes into potential themes. The visual sticky note map enabled codes to be grouped into clusters of overarching themes, and a structure of main themes and subthemes emerged, which was then translated into a Microsoft Word document together with supporting quotations. Seven members of the research team, including the two carer researchers and chief investigator, met to review the main researcher’s decisions regarding the potential themes. Discussions centred on whether or not the coded data supported each theme and the scope of each theme, with themes divided or amalgamated as needed. The descriptions given to the themes were discussed for accuracy. Discussions also explored how themes linked together and their illumination of factors affecting implementation of REACT from the participants’ perspective.

Following the meeting, the main researcher reviewed the whole data set once more to verify that the themes were robust, and the diagram was refined to illustrate the story conveyed by the data. This diagram was presented at the explanatory synthesis meeting (described in the following section) and refined further in the light of the process of integrating these data with data from other parts of the IMPART study.

Developing an explanatory synthesis and final implementation plan

Our final explanatory synthesis and IPv4 were informed by our analysis of qualitative and quantitative data from staff and relatives across all trusts, as described previously. Ideally, relatives’ data would have been collected earlier and could have informed early iterations of the implementation plan, but this was not possible.

Trust analysis days

These were held at each trust. IMPART leads invited staff and relatives who had been involved in the IMPART study to attend. The research team presented a preliminary summary of the factors affecting implementation of REACT at that trust, based on the data days analysis described in Level 2 analysis. Participants were encouraged to question, elaborate on and explore these findings, particularly highlighting any areas of misunderstanding or important missing factors. These events were a key step in ensuring that participants felt fully part of the research process, rather than just providers of data.

The IMPART study-wide data day

The IMPART leads and other key staff, including REACT champions and supporters, were invited to a 1-day event to support the interpretation of data gathered across all six trusts. The main aims of the event were to:

• present the draft final analysis, including overarching themes and key factors affecting implementation developed across the six trusts, and gather attendees’ feedback and reflections
• present all trusts with each iteration of the implementation plan and gather suggestions for IPv4
• ensure that attendees had all the information they needed about the immediate future of REACT, including how the findings of the IMPART study would be interpreted alongside those of the parallel REACT trial55
• maintain a collaborative working relationship and investigate how and if attendees wanted to continue contributing REACT in the year following the end of the IMPART study.

Explanatory synthesis event

All IMPART study team members, including members of the PMG, an IT developer, research associates and carer researchers, were invited to a final 2-day study event. The main aims were to:

• synthesise the analyses of all of the data collected throughout the study and develop a coherent story, including testing our study-specific hypotheses
• collaboratively develop an explanatory synthesis to understand the key factors that affected implementation of REACT across all six trusts
• finalise an implementation plan (version 4) that could be shared with other NHS trusts that were interested in using REACT
• reflect on the role of NPT in the IMPART study (i.e. successes and challenges) and implications for future development of the theory.

Data storage and management

All participants providing individual or group interview data gave written consent for interviews to be audio-recorded, with the following exceptions. One staff member and one staff group did not wish to be recorded, but consented to handwritten notes recording key points emerging from the discussion. One participant consented to be interviewed but did not consent to any record being made. Her interview informed the reflexive diary of the research associate. All participants were made aware that their data would be anonymised following transcription, and that anonymised quotations might feature in resulting publications and presentations.

Interviews were digitally recorded using encrypted audio-recorders [Philips Pocket Memo (Philips, Guildford, UK) and Olympus DS-2400 (Olympus, Southend-on Sea, UK)], and audio files were uploaded to secure servers at Lancaster University or UCL at the earliest opportunity; original recordings were then deleted from the recorder. Audio-recordings of interviews were sent to the transcriber electronically in a zipped [using 7-zip version 18.01 (Free Software Foundation, Inc., Boston, MA, USA)] password-protected file via a secure file-sharing digital platform [Dropbox (London, UK), Zendto version 4.10 (University of Southampton, Southampton, UK)].

All relatives’ outcome measure data were collected through an online system at Liverpool CTRC and stored on secure servers physically located within access-controlled server rooms and backed up nightly to a separate physical location. All identifiable data were encrypted using a 256-bit encryption algorithm. CTRC servers were subject to penetration testing audits undertaken by University of Liverpool central IT staff.

The IMPART data are stored on a secure server at Lancaster University, which complies with relevant statutory provisions including the Data Protection Act 2018100 and the European Union General Data Protection Regulation.101 Data held on Lancaster University servers are stored in a resilient storage infrastructure that is dual housed in the university’s data centres (on site). There are multiple levels of redundancy built into these storage arrays – snapshots and backups are automated and created regularly.

University single sign-on credentials are required to access the shared network drive; the principal investigator (PI) controls access to specific folders and ensures that this is monitored regularly.