Feasibility of evaluating effectiveness

A number of data were gathered and analysed to inform the design of a future study evaluating the effectiveness of CLECC in improving compassionate care. This chapter presents findings on the feasibility of evaluating effectiveness using a CRT design, including an assessment of bias and an outcome measure performance. It also addresses the feasibility of measuring CLECC costs and benefits.

Pilot cluster randomised trial assessment of bias

Selection bias

Selection bias was assessed by reviewing the demographic characteristics for each group that was recruited. These assessments focused on the representation of a wide range of nursing staff among the nursing questionnaire respondents, and on the inclusion of older patients and patients with cognitive impairment in observations and questionnaire completion.

Nursing staff

As noted in Nursing questionnaires , nursing questionnaire respondents represented a range of ages, ethnic groups, job roles/bands and years of experience. However, the low overall response rate to the nursing questionnaires (36%) reduces the certainty that the views of the respondents represent the views of all the staff working on the participating wards.

Participation rates of older patients and patients with cognitive impairment

Observations

In 17% (i.e. 63 out of 362) of approaches the patient was assessed as not having the capacity to make the decision to take part in the research. The care of 133 patients at baseline and of 140 patients at follow-up was observed. Mean patient age was 83 years at baseline and 80 years at follow-up.

At baseline 31% (n = 41) of patients showed evidence of cognitive impairment. This figure was lower at follow-up at 19% (n = 27). Overall, 25% (i.e. 68 out of 273) of patients observed had evidence of cognitive impairment.

Patient questionnaires

Across both assessment periods, of the patients approached to complete a questionnaire, 29% (i.e. 178 out of 621) had cognitive impairment. Of these, 117 (66%) were excluded. The most common reasons for excluding patients once approached were recorded as ‘very cognitively impaired’ (n = 44, 38% of 117 excluded) and ‘no capacity’ (n = 59, 50%). Of those patients with cognitive impairment who were assessed following approach as eligible for inclusion, 70% (i.e 43 out of 61) consented to take part and, of these, 98% (i.e. 42 out of 43) returned a completed questionnaire (Table 26).

Of the patients approached to complete a questionnaire at follow-up (baseline data not collected), 86% (i.e. 300 out of 347) were aged 65 years or older. Reasons for excluding these patients once approached were recorded as ‘no capacity’ (48%, i.e. 48 out of 101) and ‘very cognitively impaired’ (32%, n = 32). Of those assessed as eligible for inclusion, 79% (i.e. 156 out of 197) consented to take part and, of these, 91% (i.e. 142 out of 156) returned a completed questionnaire.

Detection bias

At follow-up it was not possible to conceal the allocation from nurses completing the questionnaires because they knew whether or not their ward had taken part in the intervention.

For follow-up observations, a team of researchers (n = 8) who were otherwise not connected with the study were recruited and trained. They were not directly informed of ward allocation by the research team, although two reported that they learned the ward allocation from ward staff during data collection.

Researchers involved in distributing and helping patients and visitors with questionnaire completion were not blinded to ward allocation. Patients and visitors completing questionnaires were invited to speculate on whether or not they were on an intervention ward. At follow-up, 82% (i.e. 28 out of 34) of the control ward patients asked to reply ‘yes’ or ‘no’ thought that they were on an intervention ward, compared with the 64% (i.e. 55 out of 86) of intervention ward patients who correctly identified that they were on an intervention ward. Researcher field notes record that patients found this questionnaire item confusing and that researchers often had to reword it to aid understanding. For visitors at follow-up (who were also offered ‘don’t know’ as an option), 16% (i.e. 5 out of 31) of control ward visitors and 35% (i.e. 18 out of 51) of intervention ward visitors thought that they were on an intervention ward. Most visitors (61% on control wards and 59% on intervention wards) said they did not know. This method of assessing blinding may not be valid but these findings suggest that study ward allocation was mostly concealed to patients and visitors completing questionnaires.

Contamination

One concern with interventions, such as CLECC, that aim for behavioural change, including collaborative behaviours, especially when intervention and control conditions are operating in parallel in the same organisation, is whether or not intervention practices ‘contaminate’ control ward practices. A comparison of staff names on each ward between baseline and follow-up did not reflect that any intervention ward staff had gone to work on either of the control wards. This finding, however, does not preclude the possibility that intervention ward staff joined the control ward and then left again before follow-up data collection. Researcher field notes reflect a conversation with a matron who oversaw an intervention and a control ward. The intervention ward leaders had reported the value of the CLECC cluster discussions and at a meeting of the ward leaders from that specialty (that included a control ward leader), a decision was taken to adopt cluster discussions across all of the wards, with the exception of the control ward. There was no evidence that the control ward went on to adopt these practices anyway but the possibility cannot be excluded. In the other hospital, intervention and control wards were managed in different specialties and so similar mechanisms would not be in place, but in this hospital the director of nursing visited the intervention wards and reportedly expressed an interest in extending the CLECC intervention to other wards in the hospital. Thus, although no evidence was found of contamination, we did identify pathways within organisations through which this could happen.

Pilot cluster randomised trial outcome measure performance

This section reports on the performance of the three core outcome measures: (1) quality of staff–patient interaction using QuIS observations, (2) nursing staff self-rating of empathy using JSE in the nursing written survey and (3) patient evaluation of emotional care using the PEECH instrument in the patient written survey. It also reports on the performance of the EQ-5D-5L in measuring benefit as part of the economic evaluation.

Quality of Interactions Schedule

Using observations as a method of collecting data appeared to be highly acceptable to patients, staff and visitors. As discussed in Observations , a high proportion of patients approached agreed to take part. Staff and visitor consent was not explicitly sought, although they were invited through written study information to raise objections if they did not wish a planned observation to go ahead or if they wished for the observation to be halted. Patients were also informed that they could halt the observations at any time. No planned observations were cancelled or halted because objections were raised. Although inviting staff or visitors more explicitly to share their views may have led to some sessions not going ahead, no data were gathered that indicated this.

All but one of the 120 planned observation sessions went ahead as scheduled. On one occasion, an evening observation could not proceed because the researcher could not gain access to the locked office containing the computer tablet needed for data collection and so that session was rescheduled for the same time of day the following week. Observers were able to observe staff–patient interactions of up to four patients simultaneously. On some occasions (11%, i.e. 27 out of 273 patients), individual patients were observed for less than the planned 2 hours because they left the ward during the session and the protocol dictated that researchers did not follow them. If they returned to the ward during the 2-hour period, observation resumed. Data were gathered from Monday to Friday, from 08.00 to 22.00.

At the close of each observation session, researchers were asked to record if staff being observed reported changing their behaviour because of the observation. Of 120 observations sessions, researchers did not ask staff on 29% (n = 35) of occasions, recorded that staff reported changing their behaviour on 4% (n = 5) of occasions and reported staff not changing their behaviour on 67% (n = 80) of occasions.

In a separate exercise to the main data collection, inter-rater reliability was tested over six 2-hour observation sessions, involving three researchers. Each session included two of the three researchers. The nature of the QI Tool software design meant that it was possible to calculate the reliability of the number of interactions recorded by each observer in each session, but not the reliability of the quality of interaction rating. The ICC for the number of interactions recorded per observation session was 0.93 (95% CI 0.607 to 0.990; p = 0.001), indicating high reliability. In earlier feasibility testing in acute hospital settings similar to those in the main study, using manual methods for recording quality of interaction, we found close agreement between observers in relation to the number of interactions observed (ICC = 0.97) and moderate to substantial agreement on the quality of interactions (absolute agreement 73%, κ = 0.53–0.62, depending on weighting scheme).99,122

Because main study observation and questionnaire data were gathered from different patient groups, it was not possible to test the validity of QuIS ratings against patient-reported experience. In our earlier feasibility work, however, 17 patients who had been observed using QuIS were asked to rate interactions, and 18 patients were able to complete a shortened version of PEECH. Patients without the capacity to consent were excluded. We found 79% agreement (weighted κ 0.40: p < 0.001; indicating fair agreement) between patients and QuIS observers over whether interactions were positive, negative or neutral.99 This earlier work also found a significant correlation between the percentage of QuIS interactions that were rated positively and patient responses to the individual PEECH item ‘exceeded expectations’ on the personal value subscale (Spearman’s r = 0.603; p = 0.008).99 We found a moderate (but not statistically significant) association between the percentage of positively rated interactions and PEECH’s ‘facial expression’ (Spearman’s r = 0.426; p = 0.088) and ‘social conversation’ (Spearman’s r = 0.402; p = 0.098).99

The QI Tool software performed well, enabling the accurate collection and transfer of interaction and contextual data.

Jefferson Scale of Empathy

The JSE was administered as part of the nursing survey. The whole questionnaire (not just the JSE) took a mean of 37 minutes to complete (SD = 50 minutes, minimum to maximum = 4 to 400 minutes. Completion time was calculated from inviting respondents to record the time at which they started the questionnaire and the time at which they finished. The longer completion times experienced by some individuals may have been because they were undertaking other activities rather than focusing solely on the questionnaire, but we did not gather any data to substantiate this. Completed questionnaires did not reflect that respondents had any difficulties completing them and there was no feedback about problems from ward staff. However, there was a perception that they were lengthy to complete and, therefore, hard for staff to find the time to complete them, and this may have affected response rates.

Patient Evaluation of Emotional Care during Hospitalisation

The PEECH instrument was administered as part of the patient questionnaire survey and, as noted in Patient questionnaires , had high acceptability with patients. Researchers helped patients to complete the survey in 68% (n = 242) of cases. The whole questionnaire (not just the PEECH) took a mean of 27 minutes (SD = 27 minutes, minimum to maximum = 4 to 330 minutes) and researcher field notes reflect that some patients found it too long. Field notes also reflect that having multiple response options presented by the PEECH’s four-point Likert-type scale was confusing, although this was improved when researchers presented a separate board with the responses written in larger type and invited patients to point to their chosen response. Patients completed their questionnaire in or by their bed because many had mobility difficulties and it was not practical for researchers to relocate them for the purposes of questionnaire completion. Because researcher help was required for most patients to complete their questionnaires, this location meant that answers could not be given without the risk of being overheard by others and researcher field notes reflect that some patients seemed concerned by this. Giving patients the option of pointing to their preferred response rather than verbalising it seemed to alleviate these concerns. Researchers reported that some patients had difficulty understanding some of the questions and so researchers had to reword them in order to get a response.

EuroQol-5 Dimensions, five-level version, health status

The EQ-5D-5L was administered as part of the patient questionnaire survey to inform the economic evaluation. The instrument was initially located on the patient questionnaire, before the last item which asked for demographic details, but researchers found that starting the interview with asking about demographic details relaxed patients and was a good way to assess mental capacity. As a result, the EQ-5D-5L was often the last part of the patient questionnaire to be completed when researchers were helping with questionnaire completion.

Researcher field notes reflect that patients had difficulties understanding the EQ-5D-5L measure, especially the visual analogue scale, which requires subjects to mark on a numbered scale (0 to 100) how good their health is today. These concerns are reflected to an extent in the analysis of missing EQ-5D-5L data shown in Table 27 . In total, 89% (i.e. 150 out of 168) of baseline patients and 76% (i.e. 141 out of 186) of follow-up patients gave responses in all five domains of the EQ-5D-5L.

The EQ-5D-5L scores ranged widely among patients, from 0.39 to 1 as shown in Figure 8 . A total of 13% (i.e. 19 out of 149) of patients had a negative EQ-5D-5L score (< 0). There was no strong correlation between EQ-5D-5L score and age or gender.

FIGURE 8 Scatterplot of EQ-5D-5L scores (at baseline and follow-up). (Figure not reproduced here; see the original article.)

Mean EQ-5D-5L index values across six wards and two experimental groups ( Tables 28 and 29 ) are lower than general population by age (0.75 for age > 75 years). A lower index value represents a lower quality of life, with 1 representing the best health possible and 0 representing death. Lower than average values in this population are expected given that respondents were in hospital. The EQ-5D-5L score (0.412) in the intervention group was lower than that in the control group (0.502), but there was no statistically significant difference in EQ-5D-5L score between both groups (mean 0.09, 95% CI –0.026 to 0.205). EQ-5D-5L scores were different at follow-up. Given that different patients were involved at each point, these changes, which were both positive and negative, cannot be interpreted as changes attributable to CLECC. Only if the different groups of patients in each ward had the same ailments and severity at baseline and follow-up could changes be attributed to CLECC.

The results of the QALY analysis based on the EQ-5D-5L data showed large differences between wards at baseline and follow-up. This was not surprising given that different patients with different ailments and severity were involved in each ward at baseline and follow-up. Consequently, although patients could be classified into Healthcare Resource Groups (HRGs; the basis on which NHS hospitals are paid), this does not include any measure of severity. Data on the quality of life of patients would need to be adjusted for severity if changes in EQ-5D-5L were to be interpretable. No agreed severity measurement exists and none is routinely collected in the NHS. Apart from diagnosis, procedure and demographic data, no other relevant data are collected. As a severity casemix measure was not used in this study, it is not possible to explain the differences in QALYs that were found.

Feasibility of estimating costs of Creating Learning Environments for Compassionate Care

Initial training costs were feasible to calculate and primarily focused on the cost of ward staff time attending the study day and the cost of PDN time to facilitate the implementation period (Table 30).

Future calculations of cost-effectiveness will depend on the length of time for which any benefits of CLECC are sustained beyond the initial investment of these training costs. If, for instance, the training costs are one-off costs with the effects sustained over an infinite period of time these costs become negligible when assessing cost-effectiveness and only ongoing implementation costs are relevant over time. Qualitative findings suggest that CLECC practices can be sustained for up to 12 months beyond the start of the implementation period. The cost of attaining and sustaining the benefits of CLECC over a 12-month period are therefore £53 per day per ward (£19,503/365), or £2 per bed-day (based on 30 beds per ward) [£19,503/(365 × 30)]. Were the benefits sustained over a 24-month period with no further training required, these costs would be halved.

In terms of ongoing costs of implementation, the results from qualitative interview data indicated that nursing staff implemented CLECC cluster discussions on the ward in short 5- to 10-minute sessions. Those who commented on whether or not extra time needed to be scheduled for such sessions suggested that it could be met within existing time schedules. Sessions remained within the short 5- to 10-minute slots, which was not seen as onerous. When wards became particularly busy, the sessions did not take place. CLECC sessions were popular with all the nursing staff interviewed. CLECC ‘champions’ were seen as important in ensuring that sessions took place. The nursing staff interviewed reported that the implementation of CLECC did not require additional nurse time. They considered that ward leadership was more important to ensure that cluster discussions happened, especially at busy times.

Chapter summary

This chapter has reported a number of important findings that can inform the design of a future evaluation. Findings indicate no selection bias inherent in our study recruitment processes and methods. The methods employed were inclusive of all staff levels and also inclusive of older patients with a range of cognitive abilities, ensuring excellent representation from a traditionally hard-to-reach group who are often excluded from research but who are prone to more negative experiences of hospital care. There was some evidence that observer researchers could find out the experimental group of the wards on which they were observing and this will need careful attention in a future trial. In addition, the findings of pathways for contamination beyond the intervention wards mean that any future trial design will need to avoid running intervention and control conditions in the same organisation at the same time. A more positive interpretation of these same findings is that the CLECC intervention appears to have the potential for impact beyond the target ward teams and future evaluations should aim to explore this potential.

The QuIS tool was highly acceptable to patients recruited and had the highest rate of participation by patients with cognitive impairment of all methods. We demonstrated that our recruitment and data collection plans for QuIS use were feasible in busy ward environments. The validity and reliability of QuIS was acceptable. The QI Tool software performed well, enabling straightforward data collection and upload.

Patient questionnaires, despite being very acceptable to patients, had a lower participation rate than QuIS for people with cognitive impairment. Most patients needed researcher help and there were complaints that it was too long. The PEECH and EQ-5D-5L scales were hard for some patients to use.

The response rate to nursing questionnaires was very low, particularly at the follow-up stage on the intervention wards, perhaps owing to research fatigue. There was a perception that questionnaires were lengthy to complete and that staff were too busy. The low response rate means reduced certainty that the responses to the questionnaire represent the views of the nursing staff as a whole.

A patient outcome measure, such as EQ-5D-5L, does not appear to be feasible to use to evaluate value for money for two reasons. First, given that some two-thirds of patients needed help to complete the questionnaires, it could be used in a full study only as a proxy measure. Second, and more importantly, interpretation of EQ-5D-5L scores as measures of health improvement is not possible as different patients, with different ailments and severity were involved at baseline and at follow-up. Without an adjustment for the various ailments and their severity, differences in EQ-5D-5L scores cannot be interpreted.

Through qualitative interviews with staff, we were able to establish that, aside from initial CLECC training costs (cost of staff time at team study day and cost of employing CLECC PDN), the implementation of concrete CLECC activities by ward teams was not associated with additional resource use. Nursing staff reported that the CLECC cluster discussions typically took 5–10 minutes each shift. They were dropped when wards became too busy to have them. We were also able to identify other candidate activities associated with supporting CLECC that merit attention in a future evaluation.