Workstream 4: assess cost-effectiveness and sustainability and enable implementation

Aim

Evaluate the intervention’s cost-effectiveness compared with usual care, evaluate the sustainability of changes observed in WS3, understand virtual SSME implementation and develop an implementation toolkit.

Assess cost-effectiveness

The first health economic section reports implementation costs. The second reports the modelling of costs and effects over the lifetime of patients. A summary follows, with details in Appendix 2, including costs (see Appendix 2, Tables 6–8), Appendix 3, including costs (see Appendix 3, Table 9), expected value of information (see Appendix 3, Table 10), cost-effectiveness planes (see Appendix 3, Figures 6 and 7) and meta-analysis results (see Appendix 3, Figures 8–13), model results (see Appendix 3, Table 11) and cost-effectiveness acceptability curve (see Appendix 3, Figures 14 and 15)].

Estimate implementation costs

Methods

Data collection

Embedders recorded all embedding-related activities in ‘trackers’ (Excel spreadsheet). Unit costs were applied to this measure of resource use to estimate intervention costs. Estimated costs were considered to be the incremental costs of embedding activities.

Analysis

The primary (base-case) analysis estimated embedding activity costs across ITT practices (n = 64) within the RCT (33 immediate; 31 wait-list). Secondary analyses used the 57 complete-case practices (ITT practices that had not withdrawn or missed a data extraction; 28 immediate; 29 wait-list) and separated randomisation groups (wait-list and immediate) to explore implementation length.

Costing the intervention

Intervention costs fell into six categories: embedder costs; practice costs (staff); provider costs (staff); toolkit costs (initial and recurrent); marketing materials costs; travel and subsistence. Staff costs were for the financial year 2019–20, with unit costs from published figures.32 Non-staff costs were taken from actual study expenditure and adjusted for inflation where appropriate. Initial toolkit costs were annuitised at a rate of 3.5% assuming a lifetime of 5 years for the website. As the costs for each practice occurred within 1 year of the intervention going live, no discounting was applied.

Trackers were available for all ITT practices. The only missing data were for time spent on some embedding-related activities (< 3%), for which the mean value for that activity from complete trackers was imputed.

Sensitivity analysis

The principal area of uncertainty for embedding costs was how it would be rolled out in standard practice, as opposed to within a research project. Sensitivity analyses accounted for this by assuming that: (1) embedders would be based more locally to the SSME provider and (2) the lead embedder would have a lower band as they would not need considerable research experience.

Results

Mean time from the first to last embedding activity was longer in immediate (204 days) than wait-list (134 days) practices; costs were adjusted to the longer period for comparability. The cost/patient for the ITT and complete-case practices for the scenario analysis was £0.521 and £0.585, respectively. In the sensitivity analyses, the cost/patient for the ITT and complete-case practices was £0.363 and £0.404, respectively.

Embedder costs amounted to 71% of total intervention costs, demonstrating their central role to intervention delivery. The next most significant costs were the combined costs of provider and practice staff directly incurred for them to attend meetings and follow up on activities initiated by embedders.

Conclusion

Intervention costs were calculated, with sensitivity analyses accounting for uncertainties around real-world implementation. Costs in wait-list and immediate practices were very similar, suggesting that the Embedding Package implementation was similar across the two waves and reasonably constant over time. Consequently, for the purposes of the cost-effectiveness modelling, the pooled cost per patient of £0.521 could be used (£0.363 for sensitivity analysis as this figure corresponds to the ITT analyses).

Estimate cost-effectiveness

Methods

Approach

We used an NHS and Personal Social Services perspective and the NIHR School for Public Health Research T2DM treatment model version 2 (an individual-level simulation of diabetes-related complications for people with T2DM). We obtained our individual-level population from NICE simulation distributions.9 Discounting aligned with NICE’s 2013 methods guide with future costs and quality-adjusted life-years (QALYs) discounted at 3.5% per annum.33

Costs

Implementation costs were costs of the intervention and of SSME attendance. Intervention costs were obtained from the costing analyses in the previous section and SSME costs from Gillett et al.7 All costs were in 2019–20 prices.

Effects (base case)

Structured self-management education uptake and changes in risk factors [HbA1c, body mass index, systolic blood pressure, high- and low-density lipoprotein (HDL and LDL) cholesterol levels] were from the RCT analyses for the control and intervention conditions.26 We assumed that all individuals under the intervention condition received these mean effects. In year 1 of the model, change estimates were obtained from the RCT results, which on average occurred 13.5 months into the intervention period. Year 2 outcomes and beyond were based on the analyses with the observational follow-up period for HbA1c and SSME uptake and on the RCT analyses for the other risk factors, as these were not analysed in the observational follow-up data. We assumed that intervention effects were fully maintained for 3 years.

Scenario analyses

All analyses were probabilistic. We ran the model with 5000 patients and 2000 probabilistic analysis runs. Scenario analyses were based on uncertainties in the model:

1. Future costs and QALYs discounted at 1.5% per annum.34,35
2. Assume that all statistically insignificant secondary RCT outcomes have no effect.
3. Take change in HbA1c and uptake of SSME in year 2 onwards from RCT results.
4. Use only effect data on SSME uptake from the RCT and base effect of SSME uptake on HbA1c on meta-analysis of published studies (see Appendix 3). Effect of SSME uptake is directly given to individuals who attend SSME, rather than giving all individuals mean effects estimated from the RCT. We assumed that SSME is effective for 6 years in full with waning by the 10th year.5,36
5. Scenario 3 and 4 combined.
6. Duration of effect is assumed longer than the base case:
   1. 5 years where the effect is fully maintained and no effect afterwards.
   2. 10 years full maintenance of effect and no effect afterwards.
7. Duration of effect is assumed longer than in scenario 4:
   1. 10 years where the effect is maintained in full, with gradual waning by the end of year 15.
   2. Lifetime effect.

Outcome measures

The primary outcome was incremental cost-effectiveness ratio (ICER), which is the difference in cost between intervention and control divided by the differences in QALYs between intervention and control. If one of intervention or control was more effective and cheaper than the other, then that condition was deemed dominant and an ICER was not calculated.

Results

Base case

The Embedding Package dominated usual care (it produced more QALYs at a lower cost; see Table 2). The differences in costs and QALYs were very small. However, using the Hatswell et al.37 method for assessing whether sufficient probabilistic sensitivity analysis runs had been conducted suggested that 2000 probabilistic sensitivity analysis runs were sufficient for the stability of the results as the 95% CI around the mean incremental net monetary benefit (iNMB) did not contain £0.

The results were highly uncertain; the cost-effectiveness plane had probabilistic analysis runs scattered either side of the ICER equals £20,000 per QALY line and the intervention had just over a 50% chance of providing the most net benefit at all thresholds between £0 per QALY gained and £50,000 per QALY gained.

Value of information

The expected value of information (total health economic value of having no uncertainty in the results) at a decision-making threshold of £20,000 per QALY gained was £445 per person. The value of conducting further research into the intervention effectiveness was relatively small at £24 per person.

Scenario analyses

The intervention was dominant or had an ICER < £20,000 per QALY gained in seven of nine scenarios (see Table 2).

Updated analyses post peer review

Post peer review, we identified errors in the model code relating to the implementation of estimated glomerular filtration rate (eGFR). Revised results have been run in an updated model correcting these errors and are available in Updated results. These results are similar to the original results, with the differences in costs and QALYs being small and uncertain and the intervention being dominant or having an ICER < £20,000 per QALY gained in eight of the nine scenarios. The model used to generate these results is under a General Public Licence version 2 or later.38

Conclusion

The Embedding Package dominated usual care (it was more effective and lower cost) in the base-case analyses and in six of nine scenario analyses, with the intervention having an ICER of < £20,000 per QALY in one further scenario analysis. However, the differences in cost and QALYs between the intervention and control were very small and highly uncertain with very wide CIs, suggesting no evidence of a difference in costs and QALYs. The value of information analyses indicated the biggest driver of uncertainty were the risk functions used in the model, rather than the effectiveness estimates from the RCT.

Assess sustainability of the intervention

We planned to investigate whether improvements in SSME uptake were maintained beyond WS3 through a 12-month observational follow-up in both arms during which the study team no longer reinforced the intervention, but practices and providers could continue using it if they chose to do so (without a study-provided embedder). The same outcomes were measured as in the RCT. However, the RCT results suggested that the intervention did not improve uptake, so assessing the maintenance of effects was inappropriate.

Instead, the primary RCT analysis (mean difference between control and intervention states estimated using mixed-effects linear regression) was repeated using the RCT and observational data combined to assess whether there was a difference in HbA1c between control and intervention conditions when allowing for more time to realise changes. Similarly, analyses for the main secondary RCT outcomes (referral and attendance) were repeated with the observational data included. All analyses in this subsection were exploratory and not prespecified.

This analysis included 37,825 eligible patients in the ITT practices (20,352 wait-list; 17,473 immediate). After accounting for clustering and time effects, mean HbA1c was −0.56 mmol/mol (95% CI −0.71 to −0.42 mmol/mol; −0.05%, 95% CI −0.06% to −0.04%) lower in intervention than control conditions (p < 0.001). This result was statistically significant but not clinically significant based on the predetermined threshold in the RCT sample size calculation.

Structured self-management education referrals did not differ between treatment conditions (OR 0.97, 95% CI 0.90 to 1.04; p = 0.414). However, SSME attendance was higher during intervention than control conditions (adjusted OR 1.13, 95% CI 1.02 to 1.25; p = 0.017).

We hypothesised that this longer-term improvement may be partly due to increased uptake of MyDESMOND (digital SSME programme) during the COVID-19 pandemic. Therefore, we repeated these analyses within subgroups of providers who did and did not make MyDESMOND available. HbA1c reductions were similar whether MyDESMOND was not available [−0.81 mmol/mol, 95% CI −1.08 to −0.54 mmol/mol; −0.07%, 95% CI −0.10% to −0.05%; p < 0.001] or MyDESMOND was available (−0.45 mmol/mol, 95% CI −0.61 to −0.28 mmol/mol; −0.04%, 95% CI −0.06% to −0.03%; p < 0.001). Again, these findings were not clinically significant. Similarly, attendance improvements were similar whether MyDESMOND was not available (1.27, 95% CI 0.92 to 1.76; p = 0.149) or MyDESMOND was available (1.11, 95% CI 1.00 to 1.24; p = 0.042). MyDESMOND was therefore unlikely to be the sole driver of this improvement.

These results suggest that SSME attendance and HbA1c improved compared with control conditions over a longer time period than the main RCT results, which showed no change in SSME attendance or HbA1c. While these results are promising, they should be interpreted with caution as differences were not clinically significant and ethnographic analyses and implementation data (see Appendix 1) suggest that implementation of the Embedding Package was limited in most settings. Furthermore, the observational follow-up took place during the COVID-19 pandemic when there was a substantial shift to online T2DM SSME and global messaging about the negative impact of COVID-19 in people with diabetes; both of which likely affected SSME attendance and HbA1c levels. However, these results suggest that the Embedding Package could be effective with the inclusion of a longer period over which to realise the intervention’s impact, perhaps reflecting that system-wide change may take longer to realise than allowed for within the RCT design.

Understand implementation of virtual self-management programmes

Context

A qualitative substudy, replacing the ethnographic element of WS4, focused on SSME delivery changes, particularly the move to virtual delivery, which was accelerated by COVID-19.

Due to COVID-19, we adapted our face-to-face DESMOND programme for virtual delivery in real time, via group sessions on Microsoft Teams^® (Microsoft Corporation, Redmond, WA, USA), where educators delivered the usual face-to-face content. Other SSME programmes were similarly delivered virtually during the pandemic. This is different to MyDESMOND, which is an asynchronous, self-directed digital-based programme comprising educational materials, animations and videos on T2DM and self-management, with functions including a chat forum and ‘ask the expert’.

Aim

To understand patient- and practice-level barriers and facilitators to virtual SSME uptake and implementation.

Objectives

• Understand the impact of the COVID-19 pandemic on SSME uptake.
• Explore views and experiences of virtual SSME attendees and of online educators.

Methods and results

Data analysis of one-to-one interviews (11 staff and 19 SSME attendees from 2 providers) generated three themes:

• Need for an established infrastructure.
• Staff and user experience.
• The future of virtual SSME programmes.

Appendix 4 provides more detail including characteristics of staff (see Appendix 4, Table 12) and SSME attendee (see Appendix 4, Table 13) participants.

Conclusions

This qualitative substudy explored barriers and facilitators to virtual SSME uptake and implementation at organisational and individual levels. Theoretical domains framework (to explore behavioural aspects of delivery and attendance) and NPT (to explore structural and organisational features of SSME uptake) guided data collection.

Staff from both sites expressed information technology (IT) challenges around setting up and delivering SSME programmes, indicating that an established infrastructure is fundamental for the future of virtual SSME. One locality reported enabling support via an established ‘academy’ that was freely available to educators and provider services, which seemed to level challenges around delivery.

Preference seemed to favour face-to-face SSME; however, participants who attended virtual SSME were originally invited to a face-to-face programme before the pandemic and would not have been exposed in the same way to a virtual format. This finding reiterates results from our cross-sectional study (n > 3500), which concluded that people tend to prefer modalities they have previously experienced.39

Regardless of people’s preference, all participants stated they would recommend virtual SSME to friends and family, based on the positive experience of staff and attendees. Delivering and attending a programme remotely fits with people’s life commitments, saving time and finances on transport to, and delivery of, sessions. This was deemed one of the biggest advantages of virtual SSME, as it addresses many factors that prevent people from attending in-person programmes (e.g. depending on others for transport, cost of public transport, physical barriers, inability to take time off work or time away from family commitments).

Although the convenience factor was a huge positive for virtual SSME, staff and attendees experienced practical and logistical challenges. The ability to join a session remotely resulted in unavoidable distractions and reduced attention span, adding to educators’ challenges in delivering virtual sessions. Difficulties were also expressed around not being able to read the group’s body language or visual cues, which often aid the flow and depth of group discussions. Barriers to maintaining engagement with attendees were connectivity issues from both parties and the inability to use tangible resources, which were often key to the interactive nature of face-to-face programmes.

We adopted purposive sampling for the attendees’ cohort; however, only one person from a minority ethnic group was interviewed. The sample size was small and so this potentially reflects the demographic characteristics of the two localities.

This study offers insight into barriers and enablers to virtual SSME delivery and uptake and suggests acceptability of virtual SSME from both staff and attendees. With appropriate infrastructure and support, SSME programmes could be delivered in a blended approach, face to face or virtually, offering tailored education format and patient choice.

Enable long-term intervention implementation

We planned to generalise the Embedding Package into an implementation toolkit for widespread adoption. However, WS3 results suggest that it is inappropriate to implement the package widely in its current format. Despite this, the package showed some promise. A PDF version of the toolkit was therefore produced (see Implications for practice and lessons learnt).

Changes from workstream 4 original plan

As described above, a qualitative substudy replaced the ethnographic work and the implementation toolkit was scaled back.

Instead of using a tick-box tracker for cost-effectiveness data collection and only calculating costs for 12 practices, embedders entered detailed entries in Excel, enabling calculation of costs for all practices, making costings more robust and accurate without needing to impute. Planned interviews with SSME providers were not undertaken (see Appendix 2).