This chapter outlines the methods used for analyses of the process evaluation, pilot CRT and economic evaluation. To reiterate, the study objectives were:
- to determine the feasibility of implementing the CLECC intervention and sustaining the resulting work practices
- to inform the design of a definitive evaluation of the effectiveness of CLECC
- to inform the measurement of costs and benefits of CLECC in a definitive evaluation.
Process evaluation
Data were analysed using systematic reading, familiarisation and open coding, undertaken independently by research team members and then in collaborative data analysis workshops. Coding discussions led to the development of two coding frames: the first to enable the exploration of themes related to relational care, ward leadership and teamwork across all six wards (for the CRT element), and the second, from intervention wards only, to focus on CLECC implementation and mechanisms of impact. All qualitative interview data from nursing staff were coded against these frames, with the use of constant comparative methods enabling the generation of new categories and the comparison of data in relation to these categories.
The relational care coding frame (Figure 5) was designed to support comparisons across the wards and, so, analysis focused on summarising and describing what people had to say about relational care, leadership and teamwork.
FIGURE 5 Relational care coding frame. (Figure not reproduced here; see the original article.)
A deeper level of analysis was undertaken in relation to the CLECC implementation frame and this part of the analysis was the focus of the team’s work. There was a conscious decision to avoid prematurely ‘fitting’ the data into NPT domains, and the use of the coding frame shown in Figure 6 enabled preliminary descriptive coding in relation to implementation and mechanisms of impact. The constant comparison method was used to examine the codes generated against the framework, which was further developed as analysis progressed. Narrative data summaries and matrix/charting techniques were then used to facilitate comparison with the NPT framework and test and refine emerging theories of implementation processes.
FIGURE 6 Coding frame for analysis of CLECC implementation. Based on MRC guidance89 and Pawson.116 a, Anything external to CLECC that impedes/strengthens implementation. (Figure not reproduced here; see the original article.)
Findings from the analyses of the qualitative interview data were triangulated with qualitative data from field notes kept by the PDNs leading the delivery of CLECC and researcher field notes from observations of CLECC training activities.
Most, but not all, of the team were researchers with a nursing background and had not met the research participants prior to the study. All were involved in interviewing and/or observing the staff in the study, and in data analysis. In early stages, more than one researcher coded the same data set, with comparisons between coding decisions creating opportunities for team discussion and the development of shared approaches. The deliberate use of reflective techniques during collaborative analysis workshops enabled individual views and assumptions to be surfaced and explored with a view to enhancing the quality of analysis. NVivo11 (QSR International, Warrington, UK) was used to support preliminary coding and consistency across the team.
Pilot cluster randomised trial
Data were analysed using SPSS 22117 and the significant threshold (alpha) was set at 0.05 (two tailed). Stata 14 was used for multilevel logistic model. Exploratory data analyses were performed to check the data and identify inconsistencies. The primary purposes of tests of effectiveness were to pilot procedures for analysis and to inform effect-size calculations for a future study.
Baseline characteristics
Frequencies and percentages or mean and standard deviations (SDs) were calculated for baseline and follow-up patient and ward study participant flow, characteristics and contextual features.
Quality of Interactions Schedule data analysis
All trial analyses were carried out on an intention-to-treat basis with wards included on the basis of their planned CLECC intervention status, irrespective of the extent to which they actually adopted CLECC practices. Three approaches to the analysis of QuIS data as an outcome were used. The first analysed the proportion of interactions rated using each of the five QuIS ratings. The second calculated the proportion of all negative interactions (i.e. interactions rated as either negative protective or negative restrictive) per patient. The third used the data for the subset of patients observed for a full 2 hours and calculated the proportion of patients in this subset that experienced at least one negative interaction during the 2-hour observation session. Descriptive statistics were used to display the findings from all three approaches for baseline and follow-up assessment periods, and for intervention versus control wards. Analyses by individual ward were also performed. For the first and third approaches, differences between groups were then tested using a chi-squared test.
A three-level mixed-effects logistic regression model was fitted to investigate the effect of the CLECC intervention on the likelihood of an interaction being rated as negative (protective and restrictive combined). The individual interactions recorded between patients and staff were considered the lowest level of the model. Patient and observation session were included in the model as random effects making up the higher two levels of the model. Predictive factors were included as fixed effects and presented as odds ratios (ORs) with 95% confidence intervals (CIs), both before and after adjustment for the other predictors. Models were adjusted for baseline and ward consecutively.
In addition to tests of effectiveness, we also used baseline QuIS data to quantify and characterise staff–patient interactions and to identify the factors associated with negative interaction ratings. Details of the analysis are reported in our published paper, shown in Appendix 7 .118
Patient Evaluation of Emotional Care during Hospitalisation data analysis
The PEECH tool assesses the degree of emotional care needed by patients in four subscales: security, knowing, personal value and connection. PEECH items are scored from 0 to 3, with 3 representing the best possible score (0 = none of the time, 1 = some of the time, 2 = most of the time, 3 = all of the time). Subscale scores were calculated from the mean of items in the subscale and total PEECH score from the sum of all items when at least 75% of the items in each subscale were available. These were computed by individual ward, experimental group and assessment period. Differences in scores between groups at follow-up were tested using the Mann–Whitney U-test.
Using a different approach, subscale scores were dichotomised into either low scores (patients with an average score of ≤ 2) or high scores (average score of > 2). We calculated the frequencies of patients with low scores for each subscale by ward, experimental group and assessment period. Differences between groups were tested using a chi-squared test.
Further multivariate analyses using logistic regression were performed for findings in which a significant difference had been found. Unadjusted and adjusted models were fitted to predict the outcome with the CLECC intervention as the primary predictor. Models were adjusted for ward, baseline and patient characteristics (age, gender, ethnicity and education level) consecutively. ORs and their 95% CIs are presented.
Jefferson Scale of Empathy data analysis
Mean score, SD and range were calculated for total empathy score when at least 75% of the items were available. They are presented by individual ward, experimental group and assessment period. Differences between group means at follow-up were tested using the Mann–Whitney U-test.
Intracluster correlation
Intracluster correlation coefficients (ICC) for the three core outcome measures (QuIS, PEECH and JSE) were calculated to account for clustering by ward and observation period. ICCs for QuIS were calculated based on the proportion of patients with one or more negative QuIS interactions in a 2-hour period. ICCs were calculated using the loneway command in Stata 14.
Economic evaluation
The cost of the CLECC intervention comprised one-off training costs and ongoing implementation costs. Training costs were agreed with senior managers from participating hospitals, using standardised models for calculating the cost of staff time when relevant.
The CLECC implementation costs focused on the ward-based cluster discussions and the extent to which these required additional nursing staff time, with potential implications for nurse staffing ratios. Twenty-one transcripts from T3 qualitative interviews with intervention ward nursing staff were reviewed and annotated by James Raftery with a focus on data on resource implications of CLECC. These analyses were discussed with interviewers and with Jackie Bridges to validate emerging hypotheses as regards the implementation costs of CLECC.
To aid calculation of the benefits of CLECC in a future trial, the utility of the most used quality-adjusted life-year (QALY) outcome measures was explored, namely the EQ-5D-5L. The EQ-5D-5L was administered to patients in the relevant wards at baseline and at follow-up. Completion rates were estimated. EQ-5D-5L baseline and follow-up scores were translated to quality-of-life scores using the national tariff and compared by ward at each time point.93
Chapter summary
These chapter has outlined the various methods used for analyses of the process evaluation, pilot CRT and economic evaluation.