Cluster randomised clinical trial methods

Introduction

Following completion of phase one of the POPPI study (standardisation and refinement of the POPPI intervention, and two successfully conducted feasibility studies), the POPPI intervention was ready for evaluation in a cluster RCT.62 At the end of phase one, approval to proceed to phase two (the cluster RCT) was granted by the NIHR Health Services and Delivery Research (HSDR) programme. This chapter describes the cluster RCT methods.

Aims and objectives

The POPPI cluster RCT aimed to evaluate the clinical effectiveness and cost-effectiveness of the POPPI intervention in reducing the development of patient-reported PTSD symptom severity at 6 months post recruitment. The specific objectives were to:

• evaluate the effect of the POPPI intervention on patient-reported PTSD symptom severity and other psychological outcomes and quality of life at 6 months
• estimate, in an integrated economic evaluation, the cost-effectiveness of the POPPI intervention
• assess, in an integrated process evaluation, the fidelity, dose and reach of the implementation of the POPPI intervention, and identify important contextual factors that affected its delivery.

Trial design

The POPPI cluster RCT was a multicentre, parallel-group cluster RCT, with a baseline (pre-intervention period) and staggered roll out of the intervention. Both an integrated process evaluation and an economic evaluation were embedded. The methods for the process evaluation and economic evaluation are described in Chapters 6 and 8, respectively.

The cluster RCT was nested in the CMP, the national clinical audit of adult, general critical care units in England, Wales and Northern Ireland, and co-ordinated by ICNARC.63 Nesting the POPPI intervention in the CMP ensured an efficient design (with respect to participating units and data collection) and facilitated efficient management of the trial, including monitoring recruitment.

Setting

The setting for the cluster RCT was NHS adult, general critical care units in England, Wales and Northern Ireland.

Intervention

The POPPI intervention comprised three elements:

1. creating a therapeutic environment in critical care
2. three stress support sessions for patients identified as acutely stressed
3. a relaxation and recovery programme for patients identified as acutely stressed.

Each element was facilitated by an education package, including an online training course to train all staff in a unit to create a therapeutic environment, and a face-to-face, 3-day course to train self-selected critical care nurses (‘POPPI nurses’) to deliver the stress support sessions and the relaxation and recovery programme to patients identified as acutely stressed (see Chapters 2 and 3 for more details).

Participants

There were two levels of participation: site level and patient level. Patients were nested in sites.

Sites

The cluster RCT aimed to recruit a representative sample of 24 NHS adult, general critical care units across England, Wales and Northern Ireland. Adult, general critical care units were defined as intensive care units or combined intensive care/high-dependency units. Standalone high-dependency units and specialist critical care units (e.g. neurosciences, cardiothoracic) were not eligible for participation in the cluster RCT.

Patients

Patients admitted to participating critical care units were eligible according to the following criteria.

Inclusion criteria

• Aged ≥ 18 years.
• A duration of > 48 hours in the critical care unit.
• Receipt of level 3 critical care (for any period of time) greater than 48 hours in the critical care unit.
• A score of between + 1 and –1 on the RASS.64
• A Glasgow Coma Scale65 score of 15.
• English-speaking.
• Able to communicate orally.

Exclusion criteria

• Pre-existing chronic cognitive impairment, such as dementia.
• Pre-existing psychotic illness, such as schizophrenia.
• Pre-existing PTSD.
• Receiving end-of-life care.
• Previously recruited to the POPPI study.

Outcomes

The primary clinical effectiveness outcome was the development of patient-reported PTSD symptom severity measured used the PSS-SR57 at 6 months. The primary cost-effectiveness outcomes were incremental costs [cost-effectiveness analysis (CEA)], quality-adjusted life-years (QALYs) and net monetary benefit at 6 months.

The secondary outcomes were:

• days alive and free from sedation to day 30
• duration of critical care unit stay
• a PSS-SR score of > 18 points at 6 months66
• anxiety and depression at six months, measured using the Hospital Anxiety and Depression Scale (HADS)67
• health-related quality of life (HRQoL) at 6 months, measured using the EQ-5D-5L questionnaire.56

Procedures: site level

Recruitment of sites

A call for expressions of interest was sent via e-mail to all adult, general critical care units actively participating in the CMP by the ICNARC CTU. In addition, advertisements were also placed on ICNARC’s website and Twitter feed (Twitter, Inc., San Francisco, CA, USA).

Sites were eligible to take part in the cluster RCT if able to commit to the following:

• show that recruitment to target, timely data collection, and delivery of the POPPI intervention were feasible, via completion of a site feasibility questionnaire
• dedicate adequate resources to carry out the POPPI intervention
• agree to adhere to randomisation into either the control or the intervention group
• identify two joint PIs to lead the POPPI intervention at the site (a lead nurse and a lead doctor)
• agree, when possible, to recruit all eligible patients to the POPPI intervention and to maintain a screening and enrolment log to include reasons why eligible patients were not recruited
• agree to use the CAM-ICU68 for assessing delirium and the RASS64 for assessing sedation status for the duration of the cluster RCT
• continue active participation in the CMP.

Sites that took part in the POPPI intervention feasibility study (International Standard Randomised Controlled Trial Number ISRCTN61088114) were not eligible for selection for the cluster RCT. Stand-alone high-dependency units and specialist critical care units were excluded, along with critical care units that offered formal psychological support (diaries in the critical care unit were permitted as they were not deemed an early intervention).

Site initiation

Site initiation visits, facilitated by the trial manager (PRM), were held at each participating site prior to the commencement of patient screening. The purpose of these visits was to present the background/rationale of POPPI and to train the local teams in the trial procedures (e.g. screening, recruitment and data collection). Local PIs and research team members were required to attend the site visit. Procedures relating to delivery of the POPPI intervention were not discussed at these visits, given that these procedures would not be relevant to sites randomised to the control group.

Investigator site file

An investigator site file was provided to all participating sites at the site initiation visits. This contained all essential documents for the conduct of the trial and included the approved trial protocol, all relevant approvals (e.g. favourable ethics opinion letter), a signed copy of the clinical trial site agreement, the delegation log, copies of the approved PIS, patient consent forms and all standard operating procedures (e.g. for screening participants, for obtaining informed consent and for data collection). The site PIs and other delegated individuals were responsible for maintaining the investigator site file.

Randomisation

The 24 sites were randomly assigned to either the intervention group (n = 12) or the control group (n = 12) by the trial statistician at the ICNARC CTU, using a restricted randomisation approach to ensure balance across the groups in terms of geographical location, hospital teaching status and size of unit.69 To allow for a staggered roll-out of the POPPI intervention, the 24 sites were allocated geographically into three groups of eight. Simulations of alternative ways to balance on size of unit were performed and compared:

1. balancing on teaching status and number of beds
2. balancing on teaching status and number of level 3 admissions
3. balancing on teaching status, number of beds and number of level 3 admissions.

The best combination on the above three factors (balance on teaching status and number of level 3 admissions) was used to perform the final random allocation.

For each group of eight sites, the individual sites were randomised (4 : 4) in their second month of recruitment (during the baseline period). It was necessary to randomise on a site (cluster) rather than individual level to avoid contamination of usual care, as it would not be possible to restrict parts of the POPPI intervention (e.g. the POPPI online training) to individual patients.

Sites randomised to the intervention group were then required to identify three self-selected POPPI nurses based on the following criteria:

• registered nurse with at least 3 years critical care clinical experience
• effective communicator with patients, families, colleagues and collaborators
• able to work flexibly
• interested in improving psychological care of patients
• organised and able to manage a busy schedule.

The PIs were provided with the person specification, and decisions on POPPI nurse selection were made locally, as would be the case in usual practice.

Timeline

The 24 sites opened to recruitment in three groups of eight sites at 2-month intervals and recruited patients over a 17-month period (Figure 8). Control group sites delivered usual care for the duration of the recruitment period. Intervention group sites delivered usual care from months 1–5 (baseline period). Usual care was defined as patients receiving psychological support or treatment at the discretion of the treating clinician(s) following standard practice at their site (participating units did not offer formal psychological support to patients).

FIGURE 8 The POPPI cluster RCT schedule. Reproduced from Wulff et al.70 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. Includes minor additions and formatting changes to the original. Also reproduced with permission from JAMA 2019;321(7):665–75.71 Copyright © 2019 American Medical Association. All rights reserved. (Figure not reproduced here; see the original article.)

At month six, intervention group sites underwent a one-month transition period, during which the education package was rolled out and the intervention group sites transitioned from delivering usual care to delivering the POPPI intervention (Figure 9). The intervention was then delivered until the end of the recruitment period (intervention period).

FIGURE 9 Intervention site timeline during the transition period. Reproduced from Richards-Belle et al.62 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. Includes minor additions and formatting changes to the original. (Figure not reproduced here; see the original article.)

Delivery of the POPPI intervention at the site level

Figure 9 shows the timeline for intervention group sites during the 1-month transition period.

Intervention site initiation visits

Prior to the start of the transition period, a second site initiation visit, facilitated by the trial manager (PRM) and lead clinical investigator (DW), was held locally at each intervention site. These visits were held with the local PIs, research team, the POPPI nurses and individuals responsible for education/training. At these visits, the procedures and logistics for delivery of the POPPI intervention were discussed. Training was provided on how to screen patients using the IPAT.52 A particular focus was on identifying strategies for ensuring adherence to the intervention, including plans for ensuring that all clinical critical care staff completed the POPPI online training course and that the POPPI nurses were given sufficient time to deliver stress support sessions with patients. Changes to the recruitment and data collection procedures were also outlined.

Ahead of the 3-day training course for the POPPI nurses, the nurses were given a short pre-course theory booklet (entitled POPPI Nurse: Essential Psychological Theory and Techniques) to read before attending the course. This booklet contained an overview of the POPPI nurse role as well as the psychological theory underlying, and key techniques (e.g. normalising, psychological education) used in, the stress support sessions. A nurse training manual focusing on the three stress support sessions, and early access to the POPPI online training was also provided.

Three-day training course for the POPPI nurses

At the beginning of the transition period, all POPPI nurses attended a face-to-face, 3-day course. This was a central training course to educate the POPPI nurses in the content and delivery of the stress support sessions and relaxation and recovery programme to patients identified as acutely stressed. The course had a strong focus on skills practice. See Chapter 2 for full details of the content of the 3-day training course and associated materials. The course was repeated for each group of sites and was delivered by a psychologist, two senior nurses and a research assistant, with patient representatives in attendance to discuss their experiences of critical care and to provide feedback to nurses.

Debriefing and support for the POPPI nurses

All POPPI nurses were allocated a clinical trainer from the POPPI training team to provide debriefing and support following the 3-day training course. See Chapter 2 for full details of the content of debriefing and support. Logistically, the first debriefing and support session was to be carried out once a POPPI nurse had delivered stress support sessions to their first patient. Once all POPPI nurses at the site had delivered stress support sessions to at least one patient each, the trainer then visited the POPPI nurses in their units to offer further support. During the visit, the POPPI nurses also underwent a skills development assessment (competency testing), to ensure that they met the required standards for delivering the sessions. If necessary, further support and training was offered, prior to the delivery of further sessions with patients. The POPPI nurses continued to receive debriefing and support either via telephone calls or site visits.

In addition, monthly teleconferences were held with the POPPI training team and the POPPI nurses to facilitate peer support and share ideas across intervention group sites.

Creating a therapeutic environment

This element of the POPPI intervention was delivered at the site (cluster) level. Each intervention group site aimed to create and promote a therapeutic environment by reducing sources of stress in the unit and improving staff–patient communication. This element of the intervention was facilitated by the POPPI online training, entitled ‘Key skills in psychological care’, for all clinical critical care staff to complete. See Chapter 2 for full details on the content of the POPPI online training. The POPPI online training took ≈30 minutes to complete; it comprised five sections and included an end-of-course assessment. Local research teams enumerated all clinical critical care staff at the start of the transition period, and then monthly thereafter to ensure that new staff members were registered for the POPPI online training.

In addition, local site teams taught and modelled good communication skills and psychological care at the bedside and ensured that the POPPI intervention materials were clearly displayed (e.g. posters displaying the key messages from the POPPI online training) and distributed (e.g. pocket cards, badges) throughout the unit. Staff were encouraged to implement their own ideas for creating a more healing environment.

Transition period

Following the intervention site initiation visit and 3-day training course for POPPI nurses, the POPPI nurses and local education/research teams commenced delivery of the POPPI intervention.

During the transition period, each POPPI nurse aimed to deliver stress support sessions to at least one recruited patient, identified (using the IPAT) as being acutely stressed and at a high risk of psychological morbidity. In parallel, the POPPI nurses and local education/research teams encouraged staff in their unit to create a therapeutic environment by ensuring that all clinical critical care staff completed the POPPI online training and through other educational activities (e.g. seminars and short presentations, bedside teaching and display of materials reinforcing key messages from the POPPI online training). At the end of this transition period, the POPPI nurses underwent the skills development assessment. Following the transition period, the POPPI intervention was delivered until the end of the recruitment period.

Site management

Communication

The trial manager (PRM) and assistant trial manager (ARB), supported by the data manager (NH) and research administrator, maintained close contact with the research teams at participating sites by telephone and e-mail throughout the trial.

Teleconferences were held, initially every month, then every 2 months, with participating sites. The teleconferences acted as a forum to discuss local barriers and challenges and share best practice across sites. Summary notes and key discussion points from the teleconferences were distributed after the call.

Reports summarising screening and recruitment activity were regularly sent to participating sites as an additional prompt to identify and discuss potential issues being faced locally in the recruitment of eligible patients.

Site monitoring visits

To ensure that all sites were visited by the trial team at least once during the recruitment period, on-site routine monitoring visits were conducted at all control group sites (intervention group sites were visited for a second time ahead of the transition period; see Intervention site initiation visits). During the monitoring visit, the investigator site file was checked for completeness, patient consent forms were checked for all recruited patients and source data verification was conducted on a random sample of patient case report forms. After the visit, a report was provided by the trial monitor to the PIs summarising the documents that had been reviewed and the actions required by the site team. The site PIs were responsible for addressing the actions and reporting back to the ICNARC CTU.

Maintenance and motivation

Throughout the trial recruitment period, a weekly e-mail was sent to site teams providing an update on recruitment and a newsletter was sent quarterly. These e-mails and newsletters provided an opportunity to clarify any queries related to trial conduct, to share ideas for maximising recruitment, and to provide general updates on trial progress. In addition, to ensure awareness was maintained at participating sites, pocket cards summarising the eligibility criteria were distributed to staff members and posters were displayed in relevant areas of the unit.

Support

A 24/7 telephone support service was available to site teams for advice on screening and recruitment of patients and on delivery of the POPPI intervention.

NHS support costs

Trials in critical care are challenging and expensive to conduct. Resources are needed to enable screening and recruitment 7 days per week. In addition, because of critical illness and the treatments that are typically administered to patients in critical care units, it is essential that staff members seeking consent are experienced in assessing a patient’s mental capacity and are able to effectively communicate information about research to the patient during a stressful situation. To this end, resources equivalent to 0.24 whole-time equivalent of a band 6 research nurse for 14 months at participating sites were successfully agreed as NHS support costs at the grant application stage.

Procedures: patient level

The trial procedures for recruitment and follow-up of patients are summarised in Figure 10.

FIGURE 10 Overview of patient flow through the POPPI cluster RCT. ICU, intensive care unit. Reproduced with permission from JAMA 2019;321(7):665–75.71 Copyright © 2019 American Medical Association. All rights reserved. (Figure not reproduced here; see the original article.)

Screening

On admission to the critical care unit, all patients were added to a screening and enrolment log. Once the patient had stayed 48 hours in the critical care unit, they were screened by the local critical care research team for the following ‘stable’ criteria (i.e. those criteria unlikely to change after this time point):

• aged ≥ 18 years
• receipt of level 3 care during the first 48 hours in the critical care unit
• English-speaking
• no pre-existing –
  • chronic cognitive impairment (e.g. dementia)
  • psychotic illness (e.g. schizophrenia)
  • chronic PTSD
• not previously recruited to the POPPI study.

If the patient met all the above criteria, daily screening of the following ‘transient’ criteria (i.e. those which could fluctuate) commenced:

• current RASS64 score between +1 and –1
• current Glasgow Coma Scale65 score of 15
• not receiving end-of-life care
• currently able to communicate orally
• able to give informed consent (e.g. not deemed delirious by the CAM-ICU).68

If any of the daily screening criteria were not met, the patient would be rescreened each day until either fully meeting the criteria or being discharged from the critical care unit. Once a patient met all daily screening criteria (in addition to the stable criteria), they were approached for informed consent in the unit. To ensure full and transparent reporting for the trial, the final screening status of the patient, including reasons why patients were not recruited (e.g. patient declining the invitation to take part, the patient being excluded by the treating clinician, logistical reasons), was recorded. No patient identifiers were submitted to the ICNARC CTU on the screening and enrolment log.

For local teams to become familiar with screening for eligibility, in the 2 weeks prior to the planned start date of recruitment, a screening ‘run-in’ period was implemented during which teams practised the procedures for screening (i.e. applying the eligibility criteria to patients and completing the screening and enrolment log). Eligible patients were not recruited/approached for consent during the run-in period.

Informed consent

During the recruitment period, patients who met the eligibility criteria in the critical care unit were approached and provided with written and verbal information about the POPPI intervention by an authorised member of the local research team. See Report Supplementary Materials 1 and 2 for the PIS used during the usual care period and intervention period, respectively. Potential participants were given the opportunity to ask questions and time to discuss the trial with family or friends before making their decision. After the person seeking consent was satisfied that the information had been understood and questions had been answered, they invited potential participants to sign the consent form. In providing informed consent, participants were agreeing for the trial team to have access to their medical records for data collection and to receive a follow-up questionnaire at 6 months. In addition, participants recruited at intervention group sites from month 6 onwards (see Figure 8) were offered the option to provide consent to receive an assessment with the IPAT and subsequent stress support sessions and relaxation and recovery programme (if applicable). To minimise selection bias between the intervention group and control group sites, it was possible for a patient at an intervention group site to decline participation in the intervention but still provide consent to receive the 6-month follow-up questionnaire.

On entry into the cluster RCT, the participant’s general practitioner (GP) was informed, by letter, of their recruitment into the POPPI trial.

Delivery of the POPPI intervention at the patient level

Figure 11 shows the timeline for a patient recruited at an intervention group site from month 6 onwards.

FIGURE 11 Patient timeline during the intervention period. Reproduced from Richards-Belle et al.62 This is an Open Access article distributed in accordance with the terms of the Creative Commons Attribution (CC BY 4.0) license, which permits others to distribute, remix, adapt and build upon this work, for commercial use, provided the original work is properly cited. See: http://creativecommons.org/licenses/by/4.0/. Includes minor additions and formatting changes to the original. (Figure not reproduced here; see the original article.)

The Intensive care Psychological Assessment Tool assessment

Consenting, eligible patients at intervention group sites, during the intervention period, were assessed using the IPAT52 by a trained staff member as soon as possible, but within 48 hours of consent being provided. A patient was deemed acutely stressed and at a high risk of psychological morbidity if they scored ≥ 7 on the IPAT and was then referred, as soon as possible, to a POPPI nurse to receive the three stress support sessions and relaxation and recovery programme. Patients who scored < 7 on the IPAT continued to receive usual care as determined by the treating clinician(s). If a patient scored 5 or 6 on the IPAT, they were reassessed daily, for a maximum of 3 days, until either they left the critical care unit or the score dropped below 5.

Three stress support sessions for patients identified as acutely stressed

The three stress support sessions were to be delivered by the same POPPI nurse ideally within 1 week, with the first stress support session starting as soon as possible within 48 hours following the IPAT assessment. Each session lasted ≈30 minutes and, ideally, at least the first session was delivered in the critical care unit, although sessions could be delivered elsewhere in the hospital (e.g. general ward) if the patient moved. If the patient showed signs of distress or fatigue during the session, then the session was stopped and a new visit was arranged for a more appropriate time. The State–Trait Anxiety Inventory – Short Form (STAI-6)72 was used to assess a patient’s anxiety prior to session 1 (at baseline) and at the end of session 3. If a patient showed serious signs of distress at the end of their three sessions, their medical team was informed. See Chapter 2 for full details of the content of the stress support sessions.

After the transition period, a sample of consented, acutely stressed patients were asked to consent to their stress support sessions being audio-recorded. If a patient agreed, they were asked to sign the audio-recording consent form. This was optional and did not preclude patient participation in the trial or delivery of the stress support sessions. Audio-recordings would be reviewed by the training team to monitor fidelity of the stress support sessions delivered, and would be destroyed at the end of the trial. If a patient withdrew consent for use of their session to be audio-recorded, then the audio file would be deleted and no longer used.

Follow-up

All participants surviving to 6 months were sent a follow-up questionnaire by the ICNARC CTU at 6 months. The questionnaire contained the PSS-SR57 (primary outcome), HADS67 and EQ-5D-5L56 questionnaires and a health services questionnaire (see Report Supplementary Material 3).

The follow-up process started at 157 days post-recruitment for the 6-month follow-up to allow for the administrative processes. Participating sites were contacted monthly to confirm survival status to help ensure that questionnaires were not received by relatives or friends of deceased patients. Patients indicated as having died since leaving hospital were logged and the follow-up process ended.

Questionnaire packs were sent to participants by either post or e-mail (as requested by the participant) and a self-addressed stamped envelope and pen were included in all postal questionnaires for ease of return. Participants who requested the questionnaire by e-mail were sent an electronic version of the questionnaire – which could either be completed using Microsoft Word (Microsoft Corporation, Redmond, WA, USA) and e-mailed back to the ICNARC CTU, or printed, completed on paper and posted back to the ICNARC CTU. The cover of the questionnaire included a ‘I do not wish to complete this questionnaire’ tick box for participants to indicate if they no longer wished to complete the questionnaire.

Non-responders were telephoned 3 weeks later to check whether or not they had received the questionnaire and were given the option to complete the questionnaire over the telephone, if preferable, with a trained member of the trial team. Telephone calls were made at various times from Monday to Friday between 08:30 and 20:00 to maximise the chances of contacting the patient.

Follow-up ended on receipt of a completed (or blank) questionnaire or a questionnaire with a ticked ‘I do not wish to complete this questionnaire’ box; on notification to the ICNARC CTU by telephone or e-mail that the participant wished to withdraw from the trial; or if there was no response to telephone follow-up. For questionnaire packs returned indicating that the recipient was not known at the address, the contact details for the patient were checked with the recruiting hospital and/or GP.

For patients who were identified as being a hospital inpatient or resident in a nursing home or rehabilitation centre, the relevant institution was contacted to establish the status of the patient and the most appropriate way to proceed with follow-up. If the patient had mental capacity to consent but required assistance in reading and/or completing the questionnaire, then health-care professionals usually assisted the patient.

If a completed questionnaire received at the ICNARC CTU indicated the presence of signs of serious stress (i.e. a score of ≥ 18 on the PSS-SR), anxiety or depression (i.e. scores of > 7 on the relevant subscale of HADS), then a referral letter from the lead clinical investigator (DW) was sent to the patient’s GP or the local PIs at the recruiting site.

Safety monitoring

As the treatments considered in this cluster RCT were psychological interventions, adverse events were not expected.

Data collection

A secure, dedicated electronic case report form, hosted by ICNARC, was set up to enable trial data to be entered by staff at sites (see Report Supplementary Materials 4 and 5). The electronic case report form was only accessible to authorised users and access was approved centrally by the assistant trial manager or data manager after cross-checking the site delegation log. Each individual created a unique username and password and had access to data for patients recruited at their site only.

The data set for the POPPI intervention included the minimum data required to confirm patient eligibility, described the patient population, monitored and described delivery of the intervention (at intervention group sites only), assessed primary and secondary outcomes, and enabled linkage to the CMP and NHS Digital. Table 5 shows the patient data collection schedule, which is described below.

Eligibility

For all patients, data were collected to confirm that the patient met all of the inclusion criteria and none of the exclusion criteria.

Baseline

The following data were collected for all patients at baseline (i.e. at the point of recruitment) to enable follow-up and to describe the patient population:

• full name and address of the patient and their GP
• NHS number
• date of birth
• sex
• last physiology [temperature, systolic blood pressure, pulse rate, respiratory rate, peripheral oxygen saturation (SpO₂) and fraction of inspired oxygen (FiO₂)] results prior to consent
• HRQoL health thermometer score56 [scores relate to a participant’s health on the day of assessment and range from 0 (‘the worst health you can imagine’) to 100 (‘the best health you can imagine’)] and
• STAI-673 anxiety score (scores range from 20 to 80, with higher scores reflecting greater anxiety).

Intervention period

At intervention group sites during the intervention period only, further data were collected to monitor adherence to the intervention and to describe and cost the delivery of the intervention (see Report Supplementary Material 6). These data included:

• date/time of the IPAT52 assessments
• The IPAT52 scores
• delivery of the three stress support sessions (i.e. dates, location, duration)
• delivery of the relaxation and recovery programme (whether or not patients were given the tablet computer, whether or not patients reported use of the tablet computer as well as whether or not they were given the relaxation and recovery DVD and the Getting Well, Staying Well booklet to take home)
• STAI-673 anxiety score, following session 3.

Patients were also given the opportunity to feed back their experiences of the stress support sessions via e-mail or an online form.

During the critical care unit stay

Throughout the critical care unit stay, the following data were collected daily for all patients:

• presence of delirium as defined by the CAM-ICU68
• receipt of sedatives, anxiolytics, anaesthetics, sleep medications, antipsychotics, analgesics, antidepressants and vasoactive agents
• receipt of mechanical ventilation.

At critical care unit discharge

At the time of discharge from the critical care unit the following data were collected: date and time of discharge from, or death in, the critical care unit.

At acute hospital discharge

At the time of discharge from the acute hospital, the following data were collected: date of discharge from, or death in, the acute hospital.

Data linkage with the Case Mix Programme

Linkage of patient trial data to the CMP database provided information on the baseline characteristics of patients and subsequent admission(s) to a critical care unit following discharge from the original critical care unit.

Data for the CMP are collected by trained data collectors to precise rules and definitions. The data then undergo extensive local and central validation for completeness, illogicalities and inconsistencies prior to pooling.

Data linkage with NHS Digital

Following the signing of a data-sharing agreement between the sponsor (ICNARC) and NHS Digital, to confirm survival status, the POPPI trial data were linked to national death registrations in the Medical Research Information Service Database Administration Service, held by NHS Digital.

Data management

To ensure data completeness and accuracy, data entered by sites into the electronic case report form were regularly monitored and checked. Ongoing data entry and validation at sites were closely monitored by the data manager (NH) and any concerns were raised with the site teams.

Two levels of data validation were built into the electronic case report form. The first level was to prevent obviously erroneous data from being entered, for example entering a date of birth that occurred after the date of consent. The second level involved checks for data completeness and any unusual data entered, for example a physiological variable, such as respiratory rate, that was outside the pre-defined range. Site staff could generate data validation reports, listing all outstanding data queries, at any time via the electronic case report form. The site PIs were responsible for ensuring that all data queries were resolved.

Data received from completed 6-month follow-up questionnaires were entered centrally into a secure database at the ICNARC CTU following a standard operating procedure. All identifiable information, such as names (e.g. of patients, family members or hospital staff members), was removed. All queries relating to data entry were reviewed by two members of the trial team (ARB/NH) and any disagreement was reviewed and discussed with a third member of the trial team (PRM). To ensure accuracy, all questionnaire data entered into the database were cross-checked by a second member of the ICNARC CTU team. Any errors found were logged and corrected on the database.

Following the start of the transition period at intervention group sites, data relating to the delivery of the POPPI intervention was monitored closely to ensure protocol adherence. Reports on uptake of the POPPI online training were initially provided to sites on a weekly/biweekly basis, and then monthly, to ensure that new starters and leavers were identified and to allow continued identification of those staff members who had yet to complete the POPPI online training.

Power calculation

Pre-trial power calculation

The power calculation was completed using the approach of Hussey and Hughes59 to achieve 90% power to detect a reduction from 6 points to 3.1 points (p < 0.05) in the mean PSS-SR score at 6 months, and was based on the following assumptions:

• Mean (6) and SD (7.5) of the PSS-SR score taken from patients in the feasibility study.
• Estimated ICC of 0.138 – between-site coefficient of variation 0.5, corresponding to between-site SD of 3 (this is a conservative estimate as no multicentre data were available).60 Note that the inclusion of a baseline recruitment period means that the sample size calculation is less sensitive to the degree of clustering.59
• Treatment effect of a reduction of 2.9 points on the PSS-SR score, based on a RCI for the PSS-SR of 8.6 points,61 being observed in 40% of eligible patients in the intervention periods assessed as being acutely stressed and at a high risk of psychological morbidity using the IPAT, with 16% of recruiting patients declining the intervention.
• Harmonic mean of the number of patients completing follow-up (52 patients per site per annum – corresponding to 22 patients in a 5-month period), based on data from the CMP.

With the design and the above assumptions, the estimated total number of patients recruited (based on the CMP data) for the cluster RCT was 1914 patients from the 24 sites. It was anticipated that 438 patients would be assessed using the IPAT, of which 175 (40%) would be assessed as being acutely stressed and at a high risk of psychological morbidity, and therefore receive the stress support sessions.

Final review of assumptions in pre-trial power calculation

During recruitment, in consultation with the Trial Steering Committee (TSC) and Data Monitoring and Ethics Committee (DMEC), a review of assumptions underlying the pre-trial power calculation was conducted once outcome data were available for patients recruited during the 5-month baseline period in both intervention and control group sites. This review, undertaken using data available on 9 August 2016, identified the following re-estimations of the assumptions:

• Mean (10.3) and SD (10.8) of the PSS-SR.
• ICC of 0.087 [95% confidence interval (CI) 0 to 0.192] (with mean, SD and ICC estimated using all available data from a previous observational study,52 the feasibility study and the baseline period of the cluster RCT).
• Treatment effect of a reduction of 4.2 points on the PSS-SR – estimated by retaining the same effect size as a multiple of the within-site SD.
• Harmonic mean of the number of patients completing follow-up (30.7 per site per annum – corresponding to 12.8 in a 5-month period), estimated using observed data from the baseline period.

This review established that the planned design had an anticipated 78% power under the observed parameter estimates (allowing for uncertainty in the between-site variation, between 73% and 85% power).

Consequently, the decision was taken to extend recruitment in all sites to the end of planned recruitment for the final group of eight sites (corresponding to an harmonic mean of 16.5 patients completing follow-up per site during the intervention period, allowing for the variation from 5 months’ to 9 months’ duration across sites, see Figure 8). With this extension to recruitment, the planned design had an anticipated 85% power (allowing for uncertainty in the between-site variation, between 79% and 91% power). It was anticipated that, with this extension to recruitment, the estimated total number of patients recruited would be 1378. Recruitment continued to be monitored to ensure that ≥ 1378 patients were recruited. A final decision to extend recruitment by an additional 2 months in all sites was taken to ensure that this minimum number was achieved. A protocol amendment was implemented to reflect this review of assumptions and the extension to the recruitment period.

Statistical methods

Analysis principles

All analyses were based on the intention-to-treat (ITT) principle. Patients were analysed according to the treatment group they were randomised to, irrespective of whether or not the allocated treatment was received. The final analyses were conducted according to the published statistical analysis plan70 and using Stata^®/SE Version 14.2 (StataCorp LP, College Station, TX, USA). Multiple imputation was performed in R version 3.3.2 (The R Foundation for Statistical Computing, Vienna, Austria).

Interim analysis

As the duration of follow-up for the primary outcome (6 months) was long relative to the duration of recruitment (intervention period between 7 and 11 months), no interim analysis of effectiveness was planned.

Methods for withdrawals and missing data

All patients who provided informed consent are accounted for in the trial report. Mortality at 6 months was anticipated to be 10% and loss to follow-up for the primary outcome was anticipated to be 20% among survivors. Loss to follow-up for mortality at 6 months was anticipated to be < 1%. Patients who withdrew from the trial, those who died before 6 months and those lost to follow-up for mortality were excluded from the analysis of 6-month psychological outcomes. Patients recruited during the transition period were also excluded from the analysis. All other recruited patients were included in the primary analysis, with outcomes imputed if missing.

Loss to follow-up is reported by treatment group.

Multiple imputation was used to complete missing baseline and resource use covariates, and non- and partial responses for the PSS-SR,57 the HADS67 and the EQ-5D-5L,56 under the assumption that responses were missing at random (MAR) conditional on the observed data.74 Two-level imputation (patients nested in sites) was implemented using the ‘mice’ package in R.75 The overall score on each measure was imputed, not individual item responses. The imputation model included the following covariates:

• site-level covariates (* denotes covariates used to balance treatment allocation) –
  • teaching status of hospital (teaching, non-teaching)*
  • number of beds in the critical care unit (linear)
  • number of critical care unit admissions receiving level 3 care staying ≥ 48 hours during the pre-trial period,1 April 2014 to 31 March 2015 (linear)*
  • allocated treatment group (intervention, control)
• patient-level covariates –
  • time period (baseline, intervention) and interaction with treatment group
  • age in years (linear)
  • sex (female, male)
  • ethnicity (white, non-white)
  • quintile of the Index of Multiple Deprivation (IMD) 201576 (categorical)
  • documented (i.e. in medical records) pre-existing anxiety and/or depression prior to hospital admission (anxiety, depression, both, none)
  • planned admission to the critical care unit following elective/scheduled surgery (yes, no)
  • ICNARC Physiology Score77 from the first 24 hours following admission to the critical care unit (linear)
  • last National Early Warning Score (NEWS)78 prior to consent (linear)
  • HRQoL at time of consent, assessed as health thermometer score from 0 to 100 (linear)
  • STAI-673 at time of consent, scored from 6 to 24 (linear)
  • duration of stay in the critical care unit in days (linear)
  • number of days of delirium, as assessed by the CAM-ICU68, in the critical care unit (linear)
  • number of days receiving sedatives/anxiolytics/anaesthetics in the critical care unit (linear)
  • number of days receiving sleep medications in the critical care unit (linear)
  • receipt of benzodiazepines in the critical care unit (yes, no)
  • number of days receiving antipsychotics in the critical care unit (linear)
  • number of days receiving analgesics in the critical care unit (linear)
  • number of days receiving antidepressants in the critical care unit (linear)
  • number of days receiving vasoactive agents in the critical care unit (linear)
  • number of days receiving mechanical ventilation in the critical care unit (linear)
  • duration of stay in hospital following discharge from the critical care unit (linear)
  • adherence to intervention (binary)
  • PSS-SR57 score at 6 months (linear)
  • HADS67 anxiety score at 6 months (linear)
  • HADS67 depression score at 6 months (linear)
  • EQ-5D-5L56 score at 6 months (linear)
  • health services questionnaire costs at 6 months (linear)

Fifty imputed data sets were generated using predictive mean matching. The random number seed was set prior to sampling to ensure reproducibility of results.

To evaluate the results of the primary clinical evaluation under the assumption of missing completely at random (MCAR), the analyses were repeated using complete-case data (i.e. only those patients returning a completed questionnaire).

Multiple comparisons and multiplicity

No adjustment was made to account for multiple end points or multiple subgroups; p < 0.05 was taken to represent a statistically significant result. The results of subgroup analyses were interpreted, taking into account the number of significant findings that would have been expected by chance alone.79

Statistical analyses

Screening and recruitment

The flow of clusters through the trial is displayed in a Consolidated Standards of Reporting Trials (CONSORT) diagram (see Figure 12), based on the CONSORT extension for cluster-randomised trials.80

Descriptive statistics were calculated using data from screening and enrolment logs to describe the screening, recruitment and follow-up of patients through the trial. Patient data are summarised, as follows, across treatment groups and time periods:

• total patients admitted
• total patients not meeting stable criteria –
  • stayed in critical care unit < 48 hours
  • no level 3 care during first 48 hours in critical care unit
  • aged < 18 years
  • not English-speaking
  • previously recruited to the POPPI intervention
  • pre-existing chronic cognitive impairment
  • pre-existing psychotic illness
  • pre-existing chronic PTSD
• total patients not meeting transient criteria –
  • Glasgow Coma Scale score of < 15
  • RASS64 score not between +1 and –1
  • receiving end-of-life care
  • not able to communicate orally
  • unable to consent
• total patients for whom eligibility was unknown
• total eligible patients –
  • missed
  • other reasons not enrolled
  • approached for informed consent
• total patients approached –
  • consent declined
  • recruited.

Demographic and baseline characteristics

Baseline demographic and clinical data were summarised for the ITT population, for each of the two treatment groups in each of the two time periods. Continuous variables are summarised as mean (SD) and median (IQR) whereas categorical variables are summarised as number (%). There was no statistical testing for any of the summary measures while comparing the baseline variables between the treatment groups. The following baseline variables were compared between the two treatment groups:

• age in years
• sex (female, male)
• ethnicity (white, mixed, Asian, black, other, not stated)
• quintile of IMD 201576 (1 = least deprived, 5 = most deprived)
• documented (i.e. in medical records) pre-existing anxiety/depression (anxiety, depression, both, none)
• planned admission to the critical care unit following elective/scheduled surgery (yes, no)
• ICNARC Physiology Score77 from the first 24 hours following admission to the critical care unit
• Acute Physiology and Chronic Health Evaluation (APACHE) II score81 from the first 24 hours following admission to the critical care unit
• duration of stay in the critical care unit prior to consent
• number of days experiencing delirium, as defined by the CAM-ICU,68 in the critical care unit prior to consent
• last NEWS78 prior to consent
• STAI-673 at time of consent
• HRQoL at time of consent (health thermometer score).

Delivery of the POPPI intervention

Uptake of the POPPI intervention online training is reported for intervention sites over time, as the percentage of the enumerated critical care unit staff who had completed the training course by month against a target of > 80% completion.

Co-interventions received in the critical care unit were summarised for the ITT population, for each of the two treatment groups in each of the two time periods. Interventions received were summarised as the number (%) of patients receiving the intervention, the median (IQR) number of days over which the intervention was received (among those receiving the intervention) and the mean (SD) number of days over which the intervention was received (for all patients, including those who did not receive the intervention). There was no statistical testing for any of the summary measures while comparing the intervention variables between the treatment groups. The following categories of interventions are compared between the two treatment groups:

• sedatives/anxiolytics/anaesthetics
• sleep medications
• benzodiazepines (note that benzodiazepines are included as either sedatives/anxiolytics/anaesthetics or sleep medications, as appropriate)
• antipsychotics
• analgesics
• antidepressants
• vasoactive agents
• mechanical ventilation.

Delivery of the POPPI intervention at a patient level is summarised for patients in the intervention group during the intervention period. The following are reported for all patients:

• number (%) of patients consenting to assessment using the IPAT
• among those consenting, the number (%) of patients assessed using the IPAT
• number (%) of patients with an IPAT score of ≥ 7
• median (IQR) IPAT score.

The following are reported for patients with an IPAT score of ≥ 7:

• number (%) of patients by number of stress support sessions received (0, 1, 2, 3)
• number (%) of patients by location of delivery of the stress support sessions
• reasons for not receiving all three stress support sessions
• number of patients receiving a tablet computer (% of those receiving stress support session 1)
• number of patients reporting using the tablet computer (% of those receiving tablet computer)
• number of patients receiving the relaxation and recovery DVD and the Getting Well, Staying Well booklet (% of those receiving stress support session 2)
• mean (SD) and median (IQR) STAI-6 at time of consent and after stress support session 3 for patients receiving three stress support sessions and with STAI-6 reported at both time points.

Clinical effectiveness analysis: primary outcome

The primary analysis for the clinical evaluation examines if there is a significant difference in the mean PSS-SR57 score at 6 months between patients recruited to the intervention group and those recruited to the control group. This was performed using a generalised linear mixed model (GLMM) at the individual patient level (patients nested within sites and within treatment group/time period), including the following terms:

• Fixed effects at the site level (* denotes covariates used to balance treatment allocation) –
  • teaching status of hospital (teaching, non-teaching)*
  • number of beds in the critical care unit (linear)
  • number of critical care unit admissions receiving level 3 care staying ≥ 48 hours during the pre-trial period,1 April 2014 to 31 March 2015 (linear)*
  • allocated treatment group (intervention, control)
• Fixed effects at the patient level –
  • time period (baseline, intervention) and interaction with treatment group
  • age in years (restricted cubic splines, 4 knots)
  • sex (female, male)
  • ethnicity (white, non-white)
  • quintile of IMD 201576 (categorical)
  • documented pre-existing anxiety and/or depression prior to hospital admission (anxiety, depression, both, none)
  • planned admission to the critical care unit following elective/scheduled surgery (yes, no)
  • ICNARC Physiology Score77 from the first 24 hours following admission to the critical care unit (restricted cubic splines, 4 knots)
• Random effects (intercepts) at the site level.

The identity link (i.e. linear regression) was used as the link function for the model and a robust variance estimation82 was used to estimate the standard errors of the covariates, as it adjusts for possible deviations from the model’s assumptions. Rubin’s rules were used to combine estimates from the multiply imputed data sets. The coefficients with their 95% CIs and p-values are presented for the fixed-effect covariates whereas only the coefficients with their 95% CIs are reported for the random-effect variables. The primary effect estimate is the interaction (difference in difference) between treatment group and time period.

A secondary analysis used structural mean models with an instrumental variable of randomised allocated treatment to estimate the efficacy (adherence-adjusted causal effect) of the stress support sessions among those patients consenting to the IPAT assessment and stress support sessions, assessed as being acutely stressed and at a high risk of psychological morbidity (IPAT score of ≥ 7 points) and receiving at least two stress support sessions.83

As a post hoc secondary analysis of the primary outcome, the model was refitted including an additional site-level covariate of the natural logarithm of the standardised mortality ratio (ratio of observed deaths to predicted deaths from the ICNARC_H-2015 risk prediction model)84 from the period April 2014 to March 2015.

Clinical effectiveness analysis: secondary outcomes

Analyses of the secondary outcomes were also performed using GLMMs (like the primary outcome analysis), with identity link (i.e. linear regression) for continuous secondary outcomes (reported as difference in means with 95% CIs and p-values) and logit link (i.e. logistic regression) for binary secondary outcomes (reported as odds ratios with 95% CIs and p-values). Robust variance estimation methods were used to estimate the standard errors of the covariates in both the mixed linear and the logistic regression models.

Subgroup analyses

There were planned subgroup and interaction analyses for the cluster RCT. The a priori identified subgroups that were used for the subgroup analyses are as follows:

• age
  • quartiles
• sex
  • male versus female
• socioeconomic status – quintile of IMD 201576
  • 1 (least deprived) versus 2 versus 3 versus 4 versus 5 (most deprived)
• duration of delirium
  • patients with no CAM-ICU defined delirium versus patients with CAM-ICU defined delirium less than the overall median duration of delirium versus patients with CAM-ICU defined delirium greater than or equal to the overall median duration of delirium
• STAI-673
  • quartiles
• Surgical status
  • emergency/urgent surgery versus elective/scheduled surgery versus non-surgical
• predicted PSS-SR (heterogeneity of treatment effect), from a prediction model for the primary outcome derived using data from patients receiving usual care and adjusted for a priori important covariates (age, sex, socioeconomic status, duration of delirium, STAI-6, surgical status)85
  • quintiles
• site implementation score (from process evaluation)
  • tertiles.

The evaluation of the subgroup effect on the primary outcome was carried out using a test of interaction obtained from the GLMM.79 The GLMM contained a main effect term denoting the specific subgroup of interest, main effect terms for treatment group and time period, and all second- and third-level interaction terms. The test of whether or not the treatment effect varied across subgroups was based on the third-level interactions.

Governance, management and oversight

Ethics approval and local permissions

The POPPI cluster RCT was sponsored by ICNARC and co-ordinated by the ICNARC CTU. An ethics application was submitted to the National Research Ethics Committee South Central – Oxford B REC on 27 April 2015 and the POPPI cluster RCT received a favourable ethics opinion on 15 May 2015 (reference number: 15/SC/0287). Local NHS permissions were obtained from each participating NHS trust. A clinical trial site agreement, based on the model agreement for non-commercial research in the health service, was signed by each participating NHS trust and the sponsor (ICNARC).

Trial registration

To ensure transparency, the cluster RCT was prospectively registered with the ISRCTN registry on 15 July 2015. Registration was confirmed on 16 July 2015 (ISRCTN53448131). The NIHR CRN Portfolio details high-quality clinical research studies that are eligible for support from the NIHR CRN in England. The trial was adopted onto the NIHR CRN Portfolio on 30 April 2015 and issued the NIHR CRN Portfolio number 18940. In addition, the POPPI cluster RCT was adopted onto the Health and Care Research Wales Clinical Research Portfolio on 25 November 2015.

Trial management

The trial manager (PRM) was responsible for day-to-day management of the cluster RCT with support from the assistant trial manager (ARB). In addition, the day-to- day trial team comprised the research administrator, data manger (NH) and trial statistician (JWu). The Trial Management Group (TMG), chaired by the trial manager (PRM), was responsible for overseeing day-to-day management of the trial and comprised the chief investigator (KMR), lead clinical investigator (DW), methodological co-investigators (DAH, RG, SH and ZS), clinical/psychology co-investigators (CRB, DH, DS, JWei, JWel and MM) and day-to-day trial team (ARB, JWu and NH). The TMG met regularly throughout the cluster RCT to monitor conduct and progress and to ensure adherence to the protocol.

Patient and public involvement

The involvement of former critical care patients was crucial to the successful conduct of the cluster RCT. A group of six former patients were members of the POPPI training team and attended the 3-day training courses for the POPPI nurses to talk about their experiences of critical care and to observe and provide feedback to trainees. In addition, three patient and public representatives were members of the TSC and provided input into the conduct of the trial.

Oversight committees

Following NIHR guidelines, a TSC, with a majority of independent members, was convened to oversee the trial on behalf of the funder (NIHR) and the sponsor (ICNARC). The TSC met at least annually during the trial and comprised an independent chairperson, lay members (representing patient and public perspectives), independent clinicians and researchers (specialising in critical care and psychology), the chief investigator (KMR) and the lead clinical investigator (DW) representing the TMG.

Furthermore, an independent DMEC was convened to monitor trial data and ensure the safety of trial participants. The DMEC met at least annually during the trial, comprised two expert clinicians specialising in critical care medicine and was chaired by an experienced statistician.

Substantial amendments to the cluster randomised clinical trial

Following receipt of initial favourable opinion for the trial from the REC on 15 May 2015, four substantial amendments were submitted and received favourable opinion. These were as follows:

• Amendment 1 (approved on 11 March 2016) – following completion of the POPPI feasibility phase, the assumptions underlying the power calculation were reviewed and revised, following ratification from the DMEC (see Chapter 3 for full details of this review). To understand any potential effect of the intervention on anxiety, the patient follow-up questionnaires were revised and the HADS67 replaced the CES-D-1058 to allow both depression and anxiety to be measured. A witness signature line was added to the patient consent forms to formalise the procedure by which a patient is willing to provide informed consent but unable to physically sign the consent form (e.g. as a result of reduced dexterity). A poster was included for sites to display in relatives’ rooms. A GP reporting form was added to the GP letter to allow GPs to inform the ICNARC CTU of any patient significant psychological difficulties that they may be aware of.
• Amendment 2 (approved on 1 February 2017) – following review of outcome data from the baseline (pre-intervention) period, the assumptions underlying the power calculation were reviewed and revised, following ratification from the DMEC and TSC.
• Amendment 3 (approved on 29 March 2017) – at the recommendation of the TSC and following approval from the funder (NIHR), a £5.00 gift voucher was included for participants receiving their 6-month follow-up questionnaire as a means of potentially increasing the response rate.
• Amendment 4 (approved on 29 December 2017) – as part of the data-sharing agreement between the sponsor (ICNARC) and NHS Digital, the trial team sent a new issue of the patient newsletter to participants. The primary purpose of the newsletter was to inform participants of the data flow used by the trial team to obtain survival status information from NHS Digital, provide an update on trial progress and inform participants where they would be able to find the trial results.

Network support

To maintain the profile of the POPPI cluster RCT, regular updates on progress were provided at quarterly meetings of the NIHR CRN Critical Care Specialty Group and at local CRN meetings. In addition, updates were provided at national meetings, such as the CMP Annual Conference and the UK Critical Care Research Forum.