This chapter outlines the methods used in phase 1 and phase 2 of the evaluation. Individual chapters provide further details.
Phase 1
This phase was developed as a result of a 4-week scoping exercise including initial scoping of the literature, discussions with a small number of sites, documentary analysis and discussions with colleagues at Public Health England (PHE) and NHSEI.
Methods
Setting
In England, within health-care organisations (trusts/CCGs) who delivered remote home monitoring services for COVID-19 patients (including CO@h and CVW models). The evaluation took place between July and September 2020.
Procedure
Systematic review
A rapid systematic review of virtual ward models (led by primary and secondary care) was conducted. The systematic review followed the method proposed by Tricco et al.48 (see Chapter 3).
Implementation study
A rapid multisite implementation study which combined qualitative and quantitative approaches. Interviews were conducted with staff across eight sites in England. Additionally, we collected data on impact, costs, and staffing models (see Chapter 3).
Ethical approval
This phase received ethical approval from the University of Birmingham humanities and social sciences ethics committee (ERN_13-1085AP37) and was categorised as a service evaluation by the Health Research Authority (HRA) decision tool and University College London/University College London Hospitals (UCL/UCLH) Joint Research Office.
Phase 2
The protocol was informed by phase 1 and developed with input from our clinical advisory group (colleagues at PHE, NHSEI, NHS Digital and NHSX), and with other research teams working in this area.
Methods
Setting
In England, within health-care organisations (trusts/CCGs) that delivered remote home monitoring services for patients with COVID-19 (including CO@h and CVW models). The evaluation took place between October 2020 and November 2021 (data collection took place between February and June 2021, but the patient survey retrospectively included January 2021).
Design
A rapid multisite study combining qualitative and quantitative approaches to analyse the implementation and impact of remote home monitoring services for patients with COVID-19. The design of phase 2 built on phase 1 methods and findings and consultation with our clinical advisory patient and public involvement (PPI) groups and the 70@70 nurses. Specifically, findings from phase 1 informed focus on outcomes and patient experience, and the sampling approach.
Ethical approval
The study protocol was divided into two separate protocols:
- Covered workstreams 1 and 2 and the staff elements of workstreams 3 and 4, received ethical approval from the University of Birmingham humanities and social sciences ethics committee (ERN_13-1085AP39) and was categorised as a service evaluation by the HRA decision tool and UCL/UCLH Joint Research Office (January 2021).
- The patient experience study (survey and case study interviews – workstreams 3 and 4) was reviewed and given favourable opinion by the London-Bloomsbury Research Ethics Committee (REC reference: 21/HRA/0155: Feb 2021).
The study team was aware of the sensitive nature of this research for organisations and individuals, and has experience in conducting research on similar sensitive topics. The team maintained the independence of the research, followed an informed consent process, and maintained the anonymity of participants and organisations.
The patient experience study was categorised as an urgent public health study. The primary and local clinical research networks supported the team to get local approvals and participant identification centre agreement sign off at each site.
Theoretical lens
We analysed remote home monitoring models using different theoretical lenses. We explored remote home monitoring models within the social and political context in which they are designed and implemented (including the clinic and home), the multiple realities, assumptions and values that play a role in their implementation, the organisational structures that shape experiences of receiving and delivering care and the sociopolitical issues that frame development, diffusion and use of technology.49 This lens went beyond an analysis of remote home monitoring solely as a technological innovation to consider dimensions such as: self-management, accountability and clinical responsibility, ‘personalised care’, inequalities in access to care and ‘caring at a distance’.50–52
Furthermore, we used a framework with a sociotechnical lens that incorporates non-adoption, abandonment and challenges to the scale-up, spread, and sustainability of technologies for health and social care (see Chapter 11). This includes expected and necessary changes/adaptation to staff working practices and the context for widespread use of the technology. This framework (informed by theory and evidence) describes the barriers to successful uptake of innovations and provides a guide to the type of issues that should be considered by evaluators.53
Procedure
The evaluation consisted of four workstreams (see Figure 2).
FIGURE 2 Overview of the methods for each workstream. (Figure not reproduced here; see the original article.)
Workstream 1: the clinical effectiveness of COVID-19 remote home monitoring services
Workstream 1 aimed to use combinations of aggregate population-level data and hospital administrative data to both describe and estimate the impact of the roll-out of remote home monitoring models for CO@h and CVW services.
Study design
The study of the impact of CO@h on mortality and admissions was designed as an area-level analysis combining aggregated data from different sources. The aim was to use these data as time series to enable us to investigate ‘dose–response’ relationships between the evolving rates of enrolment to the services within each area and their outcomes, ‘enrolment rates’ being defined as the ratio of people enrolled on to services to the number of new cases each fortnight. For the in-hospital outcomes, we used an observational design relating in-hospital mortality and lengths of stay at an individual patient level to the rate of enrolment to CO@h services within the area at the time of admission.
For the analysis of impact of virtual ward services, we designed an admission-level multivariate regression analyses for patients with COVID-19 being discharged from hospital. The aim was to examine the association between various factors – including the availability of a virtual ward – with the length of COVID-19 inpatient spells, and likelihood of subsequent COVID-19 readmissions.
Setting and population
We evaluated the CO@h service across the CCG areas in England (N = 37) where we judged there was complete data on numbers of people enrolled on to services (onboarded) between 2 November 2020 and 21 February 2021. The study population included anyone aged 65 years or over with a laboratory-confirmed positive test for COVID-19 and any hospital admission within that age group for COVID-19 or suspected COVID-19.
For the virtual ward study, we extracted information from inpatient Hospital Episode Statistics (HES) data on all individuals discharged alive from any of 123 English hospital trusts where there had been a confirmed or suspected COVID-19 International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10) primary diagnosis code recorded during the stay. We included all patients discharged between 17 August 2020 and 28 February 2021, a period covering the beginning, the peak, and the start of the decline of England’s second COVID wave. Where a patient had two or more relevant inpatient stays, all stays were included in our analysis.
Data sets used
For our analysis we used data from a number of sources (see Report Supplementary Material 1).
Procedure and data analysis
For CO@h we analysed four main outcomes: mortality from COVID-19, hospital admissions for people with confirmed or suspected COVID-19, in-hospital mortality for these admissions and their lengths of stay. Because we were using population-level data, we first calculated enrolment rates for CO@h over time and then used multivariate regression models to investigate relationships between levels of enrolment and outcomes.
To analyse outcomes for patients admitted to hospital, we used individual-level HES. Again, we used multivariate regression models relating enrolment rates to outcomes, adjusting for individual patient characteristics (see Report Supplementary Material 1).
For the CVW analysis we developed multivariate regression models to examine the impact of the availability of CVW on two outcomes: (1) the LOS of the COVID-19 inpatient stay, and (2) on subsequent readmissions for COVID-19. Our models adjusted for the following person, inpatient spell, and trust characteristics: age at admission, gender, ethnic group, Charlson Comorbidity Index category,54 whether the stay was the person’s first COVID-19 hospital stay, whether the inpatient stay was an emergency admission, Index of Multiple Deprivation (IMD) decile, the time period of the discharge date, hospital trust, and a trust-and-week-specific measure giving the proportion of all acute beds occupied by patients with COVID-19.
Further details are provided in Report Supplementary Material 1.
Workstream 2: economic analysis
The aim of workstream 2 was to conduct a cost analysis of remote home monitoring that comprised an analysis of setting up and running remote home monitoring services for patients with COVID-19. As described in the study protocol, we originally planned to undertake a cost–utility analysis of remote home monitoring services for patients with COVID-19.55 However, in the absence of any impacts on mortality or hospitalisation being demonstrated by the effectiveness analysis (workstream 1), a cost–utility was considered uninformative and our analysis focused only on analysing the costs of setting up CO@h and CVW services and mean running cost per patient monitored.
Procedure and data analysis
Remote home monitoring models were costed using data on staff (clinical and non-clinical) and non-staff resource use during the second wave. Contacts at each of the 28 sites (see workstream 3/4 for further details) were asked to complete a form in Microsoft Word (Microsoft Corporation, Redmond, WA, USA) (see Project website documentation – Study instruments) and 26 sites returned it electronically to the researchers. The data collection process was conducted between April and June 2021. Information was collected on staff (including the number of staff, their function/seniority and the number of hours worked) and non-staff resources used for setting up and running the sites (including information on the type and costs of digital monitoring platform used, oximeters, other medical equipment, etc.). We also collected information regarding the number of patients involved (triaged, monitored, deteriorated, escalated and deaths). The cost analysis was conducted separately for CO@h and CVW services.
Costs of setting up remote home monitoring services for patients with COVID-19 We calculated the costs incurred for setting up and running the CO@h and CVW models, also examining the variation by whether data submission was undertaken using technology-enabled with analogue modes versus analogue-only modes. The set-up costs were reported separately as mean costs per site. Note that the lifetime of the sites extends beyond the period observed in the study (e.g. some of the equipment, such as digital platforms and thermometers, can be used for longer than the study time) and therefore we did not calculate the mean set-up costs per patient or per unit of time. The running costs were calculated as mean costs per patient monitored and included the costs of staff (clinical and non-clinical) who were involved in running the services, and non-staff costs including costs of pulse oximeters (based on the number of patients monitored and assuming that 70% of pulse oximeters would be returned and reused) and other monthly expenditures related to maintaining the digital platforms. We adjusted the resources used and costs incurred per patient by applying weights based on the number of patients monitored by each service (CO@h and CVW) and the total numbers of patients monitored at each site. To investigate the effect of the type of service, data submission mode, seniority of full-time equivalent (FTE) staff and the total number of patients monitored factors on the mean running cost, we have used an ordinary least square regression model.
Technology-enabled versus analogue-only modes To explore the differences in staff, resources, types of patients and patient outcomes further between tech-enabled and analogue versus analogue-only data submission modes, additional data were collected from four of the 28 sites. All of these four sites used both tech-enabled and analogue data submission modes. We conducted an in-depth analysis of the time spent per patient for specific activities of the programme and calculated the cost per patient for all the activities. The mean cost for each activity per patient was calculated based on the time spent for each activity and staff working on each activity.
Workstreams 3 and 4: implementation and patient and staff experiences
The aim of these workstreams was to (1) understand the development of national COVID-19 remote home monitoring services and, (2) analyse the implementation of COVID-19 remote home monitoring services, and patient and staff experiences of care.
These workstreams included data from: staff surveys and interviews, patient/carer surveys and interviews, interviews with national leaders and documentary analysis of relevant national operating procedures (see Report Supplementary Material 1 for detailed methods).
Sample and recruitment
Selection of sites Twenty-eight services were included in our national evaluation. To obtain maximum variation, we sampled services based on a range of criteria, including the setting (primary care or secondary care), type of model (pre-hospital, early discharge, both), mechanism for patient monitoring (paper-based, app, both), geographic location (across different areas of the country), timing of implementation (implemented since wave 1 of the pandemic or recently implemented) and involvement in the evaluation with the other evaluation partners (Imperial and IAU).
To conduct an in-depth analysis of implementation, staff and patient experiences, 17 of these sites were selected as in-depth case study sites using the aforementioned criteria. In four of these sites, we also conducted in-depth comparisons of technology-enabled versus analogue models (see Economic analysis above). These sites were identified by NHSX to ensure inclusion of different digital platforms.
Sites were recruited through an expression of interest process whereby we presented our study at local and national meetings and asked sites to express interest in participating. Clinical Research Networks facilitated the set-up of sites and local governance approvals. Some sites were identified through our phase 1 evaluation.
Staff (survey and interviews) We conducted a survey of staff involved in delivering COVID-19 remote home monitoring services in 28 services (including clinical leads, delivery staff and data staff). The study coordinator at participating sites distributed links to the online surveys to staff involved in delivering/leading services via e-mail.
For staff interviews, we purposively sampled staff involved in delivering services, leading services and data processes at each of the 17 sites. Potential participants were approached by staff at participating sites and were introduced to the researcher or asked to contact the researcher if they would like to take part. The researcher sent potential participants an information sheet. If they agreed to participate, they were asked to sign a consent form.
Patients and carers (survey and interviews) To take part in our evaluation, participants needed to be
- 18 or over
- proficient in English (or one of Polish, Bengali, Urdu, Punjabi, French and Portuguese)
- eligible to receive remote home monitoring services for COVID-19, and must also have been offered and received COVID-19 remote home monitoring services.
We were flexible within our sampling to consider both national and local eligibility criteria.
A total of 25 of the 28 sites agreed to run the patient survey. NHS staff from participating services sent the patient survey to patients and carers either via post, text or e-mail. Patients/carers completed the surveys and returned these to the researchers either electronically or via post.
For patient interviews, we aimed to purposively sample four to six patients or carers who received services or declined/disengaged from services at each of the 17 case study sites. We sampled using a range of criteria including age, gender, ethnicity, employment status, mechanism for onboarding, type of monitoring and outcomes. Potential participants were approached by staff at participating sites and were asked if they would be interested in taking part. Participants were then sent an information sheet by the researcher. If the patient/carer agreed to take part in the study, they were asked to sign consent forms and send them to the researcher ahead of the interview.
National lead interviews and documentary analysis To understand development of the national service and capture changes in design and implementation over time, we conducted interviews with national leaders who were purposively selected from different roles across organisations. We also analysed key documents (e.g. SOPs).
For the national lead interviews, national leads were contacted via e-mail and asked if they would like to take part. If happy to take part, they were asked to give their consent in advance of the interview. Researchers then arranged a convenient time for the interview.
Measures
Staff (survey and interviews) The staff survey was developed for this study. We developed different questions for different groups (service leads and staff delivering the service). The staff survey included questions about experiences delivering remote home monitoring services for patients with COVID-19. The questions were reviewed and piloted prior to use (see Project website documentation – Study instruments for survey questions).
We developed a topic guide for staff interviews. Different questions were developed for different groups: (1) service leads, (2) staff delivering the service, (3) data. The topic guide included questions about the service, experiences of delivering services and views on patient engagement (see Project website documentation – Study instruments for topic guide). Interviews at four of the 17 sites included think-aloud questions about using the technology.
Patients/carers (survey and interviews) The patient/carer survey was developed for this study. The survey included questions about experiences receiving remote home monitoring services for patients with COVID-19. The questions were reviewed and piloted prior to use (see Project website documentation – Study instruments for survey questions).
We developed a topic guide for patient/carer interviews. Different questions were developed for those who received the service, and those who declined or disengaged. The topic guide included questions on their experiences of receiving services and engagement (see Project website documentation – Study instruments for topic guide).
Survey and interview questions were informed by relevant service documentation,38,39 theoretical frameworks relating to social, political and technical contexts49–53 and behaviour,56 and previous literature on engagement.57,58 Questions on demographic characteristics were informed by previous literature.59–64
National leads (interviews) We developed a topic guide for national lead interviews. This included questions about the service, leadership and governance, data and implementation (see Project website documentation – Study instruments for topic guide).
Data collection
Staff (survey and interviews) Staff were asked to follow an online link to complete the survey. The survey began with an information sheet and consent form. Survey sites were asked to keep a record of the number of surveys sent to determine response rates.
Interviews were conducted by one of six postdoctoral researchers (MS, CV, HW, LH, IL, JB); each site had a different lead researcher. Interviews were carried out via telephone or an online platform as preferred by the participant.
Patients/carers (survey and interviews) Patients were asked to complete the survey in one of two ways: (1) if monitored through use of a tech-enabled mode, patients/carers were e-mailed or texted the link to the online survey, or (2) if the patient was monitored using analogue methods, they received the survey in the post, together with a pre-paid envelope. Surveys were distributed at onboarding to the remote home monitoring service or discharge from the remote home monitoring service, depending on service procedures. The survey included an information sheet and consent form.
Interviews were conducted by one of six postdoctoral researchers (MS, CV, HW, LH, IL, JB); each site had a different lead researcher. Interviews were carried out via telephone or an online platform as preferred by the participant.
National leads (interviews) National lead interviews were conducted by four researchers (MS, CV, HW, NJF). Interviews were carried out via telephone or an online platform such as Zoom (Zoom Video Communications, San Jose, CA, USA) or Microsoft Teams (MS Teams) (Microsoft Corporation, Redmond, WA, USA) as preferred by the participant.
Data management Surveys (patients and staff) Electronic surveys were transferred directly into Research Electronic Data Capture (REDCap, IBM Corporation, Armonk, NY). Surveys received via post were returned to the Nuffield Trust or UCL offices, stored securely in locked filing cabinets and inputted into REDCap by members of the research team. Data were securely stored on UCL’s Data Safe Haven portal.
Interviews (national lead, patients/carers and staff) All interviews were audio-recorded (with consent), transcribed verbatim, anonymised and stored securely.
Analysis
Site characteristics Sites were characterised with respect to their population size,65 the proportion in urban versus rural areas,66 the proportion in the most and least deprived areas (with respect to national quintiles),67 and ethnicity.68 For sites based on CCG areas we calculated these characteristics using publicly available data at lower-layer super output area (LSOA) level mapped to CCGs,69 while for trust-based sites we used data derived from inpatient HES admissions during the financial year 2019–20, in addition to web searches for the trust catchment populations.
Surveys (patients/carers and staff) Survey data were analysed using descriptive statistics, multivariate and univariate analyses on Statistical Package for the Social Sciences (SPSS, version 25). Participating sites were provided with a report on the patient survey responses from their site compared with overall findings.
Interviews (national lead, patients/carers and staff) Data collection and analysis was facilitated through the use of rapid assessment procedure (RAP) sheets.70 RAP sheets were developed per site to facilitate cross-case comparisons and per population (to make comparisons between subgroups). The categories used in the RAP sheets were based on the questions included in the interview topic guide, maintaining flexibility to add categories as the study progressed. RAP sheets were used to develop coding frameworks to explore themes and subthemes in more depth and identify example quotes. Further details on analysis methods are reported within each chapter (see Chapters 4 and 8–11).
Although our study was rapid, many steps were taken to ensure the transparency, rigour and robustness of our qualitative research methods. For example, our study used purposive sampling to recruit a large number of participants with a range of characteristics. In terms of analysis, our study used both rapid analysis methods (e.g. RAP),70 together with traditional qualitative analysis methods (e.g. in-depth coding of transcripts and development of themes and subthemes). Our study involved a range of experienced qualitative researchers, and our analysis was strengthened by continuous team discussions about the interpretation of our data. We also discussed findings with those providing the service and our clinical advisory group to sense check our interpretation. Additionally, our findings were triangulated with quantitative, economic and other qualitative analyses. Our methodological approaches are consistent with previous recommendations on conducting trustworthy, robust and rigorous qualitative research.71–74
Integration
Data from all four workstreams were integrated to answer our research questions and to facilitate processes of triangulation. The study team met weekly throughout the study. We included findings from each workstream within our RAP sheet. We produced RAP sheets at site level and also at population level (including for subgroups of staff and patients). For example, findings on local barriers and facilitators to implementation and patient and staff experiences helped the interpretation of findings on outcomes, service use and costs. Quantitative data on resource allocation were understood in relation to qualitative data on staff experiences of planning and delivering services. Data from the case studies were used to explain the survey findings (representing experiences and trends at a national scale). In addition, emerging findings from this study were discussed in relation to those from the two other evaluation partners (Imperial and IAU).
Deviations from protocol
We planned to undertake a cost–utility analysis of the COVID-19 remote home monitoring services but in the absence of any impacts on mortality or hospitalisation, as demonstrated by the effectiveness analysis (see Chapters 5 and 6), the economic analysis only included the costs of setting up and running services (see Chapter 7).
We intended to conduct ‘think aloud’ sessions with patients who had used tech-enabled data submission from the four sites selected for in-depth analysis of tech-enabled platforms. However, patients did not have access to the platforms after discharge and recall of the use of these platforms during their illness was poor. Consequently, the ‘think aloud’ methodology was discontinued for patient interviews.
Patient and public involvement
Members of the study team met with service user and public members of the BRACE PPI group and health and care panel and patient representatives from RSET in a series of workshops. These workshops informed study design, data collection tools, data interpretation and to dissemination for phase 2. For example, patient-facing documents, such as consent forms, topic guides, patient survey and patient information sheets were reviewed by this group. Additionally, PPI members helped to pilot patient surveys and interview guides with the research team. We also asked some members of the public to pilot the patient survey.
Dissemination and formative feedback
Throughout the project, we regularly shared feedback with stakeholders on emerging findings and lessons learned. The study will inform clinical practice for COVID-19 but will also inform the use of remote monitoring models in other conditions and areas of the NHS. Dissemination has been facilitated through various existing and new networks. Findings have been and will be published in peer-reviewed journals.