Review methods

Some parts of this chapter have been reproduced from Chambers et al.1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The text includes minor additions and formatting changes to the original text.

This systematic review was commissioned to provide NHS England with an independent review of previous research in this area to inform strategic decision-making and service design. The research questions that the review addressed are as follows (in relation to digital and online symptom checkers and health advice/triage services):

• Are these services safe and clinically effective? Do they accurately identify both patients with low-acuity problems and patients with high-acuity problems?
• What is the overall impact of these services on health-care demand? In particular, is there evidence that they drive patients towards higher-acuity services (e.g. ambulance and ED use)?
• Do individuals comply with these services and the advice received? Are these services used instead of, or as well as, other elements of urgent and emergency care?
• Are these services cost-effective?
• Are patients/carers satisfied with the services and the advice received?
• What are the implications of these services for equality of access to health services and inclusion/exclusion of disadvantaged groups?

Literature search and screening

Initial scoping searches revealed that a highly sensitive search strategy (designed to retrieve all relevant items), typically conducted for systematic reviews, retrieved a disproportionately high number of references on GPs’ decision-making and triage. These references are outside the scope of the NHS Digital 111 review and would have unnecessarily diverted resources from the review focus. Therefore, to optimise retrieval and sifting we devised a three-stage retrieval strategy as an acceptable alternative to comprehensive topic-based searching. This involves:

1. Targeted searches of precise, high-specificity terms (designed to limit the search to items likely to be relevant). Box 1 shows an example for MEDLINE. We searched the following databases –MEDLINE via OvidEMBASE via OvidThe Cochrane Library via Wiley Online LibraryCINAHL (Cumulative Index to Nursing and Allied Health Literature) via EBSCOhostHMIC (Health Management Information Consortium) via OpenAthensWeb of Science (Science Citation Index and Social Science Citation Index) via the Web of Knowledge at the Institute for Scientific Information, now maintained by Clarivate Analyticsthe Association of Computing Machinery (ACM) Digital Library.The searches were not restricted by language or date.
2. Phrase searching for named and generic systems. The list below was used for phrase searching and was compiled from a study of symptom checker performance6 and an examination of internet search results for symptom checkers –AskmdaskmygpBetterMedicineDocResponseDoctor DiagnoseDrugs.comEarlyDocEconsultengage consultEsagilFamily DoctorFreeMDgp at handHarvard Medical School Family Health GuidehealthdirectHealthlineHealthwiseHealthy ChildrenIsabeliTriageMayo ClinicMEDoctorNHS Symptom Checkersonline triagepush doctorSteps2CareSymcatSymptifySymptomatewebgpWebMD.The phrase searches were conducted on the databases listed in point 1, with no language or date restrictions.
3. Citation searches of key included studies and reviews. Reference checking of included studies and key reviews. This was complemented by contact with service providers, directly and via websites.

Search results were stored in a reference management system (EndNote; Clarivate Analytics, Philadelphia, PA, USA) and imported into EPPI-Reviewer software for screening (version 4; Evidence for Policy and Practice Information and Co-ordinating Centre, University of London, London, UK). The search results were screened against the inclusion criteria by one reviewer, with a 10% sample screened by a second reviewer. Uncertainties were resolved by discussion among the review team.

Inclusion and exclusion criteria

Population

Members of the general population, which includes adults and children, who are looking for information online or digitally to address an urgent health problem, which includes issues arising from both acute illness and long-term chronic illness. Non-urgent problems, such as possible Asperger syndrome or memory loss/early dementia, were excluded.

Intervention

Any online or digital service designed to assess symptoms, provide health advice and direct patients to appropriate services. This reflects the role of the NHS111 telephone service. Services that provide only health advice were excluded, as were those that offer treatment (e.g. online CBT services).

Comparator

The ‘gold standard’ comparator is the current practice of telephone assessment (e.g. NHS111) or face-to-face assessment (e.g. general practice, urgent-care centre or ED). However, studies with other relevant comparators (e.g. comparative performance in tests or simulations) or with no comparator were included if they addressed the research questions.

Outcomes

The main outcomes of interest were:

• safety (e.g. any evidence of adverse events arising from following or ignoring advice from online/digital services)
• clinical effectiveness (any evidence of clinical outcomes associated with use of online/digital services)
• cost-effectiveness (including costs and resource use)
• accuracy – this refers to the ability to provide a correct diagnosis and distinguish between high and low acuity/urgency problems (and hence direct patients to appropriate services, avoiding over- or undertriage)
• impact on service use/diversion (including possible multiple contacts with health services)
• compliance with advice received
• patient/carer satisfaction
• equity and inclusion (e.g. barriers to access, characteristics of patients using the service compared with the general population).

This list is not exhaustive and other relevant outcomes from included studies were extracted.

Study design

We did not restrict inclusion by study design (and included relevant audits or service evaluations in addition to formal research studies), but included studies had to evaluate (quantitatively or qualitatively) some aspect of an online/digital service. Studies from any high-income country’s health-care system were eligible for inclusion.

Excluded

The following types of studies were excluded from the review:

• studies that merely describe services without providing any quantitative or qualitative outcome data
• conceptual papers and projections of possible future developments
• studies conducted in low- or middle-income countries’ health-care systems.

Data extraction and quality/strength of evidence assessment

We extracted and tabulated key data from the included studies, including study design, population/setting, results and key limitations. The full data extraction template is provided in Appendix 1. Data extraction was performed by one reviewer, with a 10% sample checked for accuracy and consistency by another reviewer.

To characterise the included digital and online systems as interventions, we identified studies reporting on a particular system and extracted data from all relevant studies using a modification of the Template for Intervention Description and Replication (TIDieR) checklist8 that we designated Template for Intervention Description for Systems for Triage (TIDieST). The checklist is presented in Appendix 2 and the completed checklists in Appendix 3. Characteristics of included systems are summarised in Chapter 3, Characteristics of included systems.

Quality (risk-of-bias) assessment was undertaken for peer-reviewed full publications only (i.e. not grey literature publications or conference abstracts). The rationale for this approach was that non-peer-reviewed publications tend to lack the detail required for assessment of risk of bias and/or tend not to follow standard study designs. Randomised controlled trials (RCTs) were assessed using the Cochrane Collaboration risk-of-bias tool.9 For diagnostic accuracy type studies, we used the Cochrane Collaboration version of Quality Assessment of Diagnostic Accuracy Studies (QUADAS)10 and for other study designs we used the National Heart, Lung and Blood Institute’s tool for observational cohort and cross-sectional studies.11 Details of quality assessment tools can be found in Appendix 4. Quality assessment was performed by one reviewer, with a 10% sample checked for accuracy and consistency by another reviewer. Assessment of the overall strength (quality and relevance) of evidence for each research question is part of the narrative synthesis. Overall strength of the evidence base for key outcomes was assessed using an adaptation of the method described by Baxter et al.12 This involves classifying evidence as ‘stronger’, ‘weaker’, ‘inconsistent’ or ‘very limited’, based on study numbers and design. Specifically, ‘stronger evidence’ represented generally consistent findings in multiple studies with a comparator group design or comparative diagnostic accuracy studies, ‘weaker evidence’ represented generally consistent findings in one study with a comparator group design and several non-comparator studies or multiple non-comparator studies, ‘very limited evidence’ represented an outcome reported by a single study and, finally, ‘inconsistent evidence’ represented an outcome for which < 75% of the studies agreed on the direction of effect. All studies included in the review were included in the analysis of the overall strength of evidence.

Evidence synthesis

We performed a narrative synthesis structured around the prespecified research questions and outcomes. This included an ‘evidence map’ summarising the quantity and strength of evidence for each outcome and identifying gaps that may need to be filled by further research. We did not perform any meta-analyses because the included studies varied widely in terms of design, methodology and outcomes.

Public and patient involvement in the study

The study aimed to be informed by patient and public involvement (PPI) at all stages of the research process. The Sheffield Health Services and Delivery Research (HSDR) evidence synthesis centre patient and public advisory group provided input during the design, analysis and reporting phases, including exploration of the study parameters, discussion regarding the meaning and interpretation of the study findings, drafting of the Plain English summary and help with disseminating the findings and maximising the impact of the research.

The advisory group comprised nine members drawn from the Yorkshire and Humber region and two members from other regions of England. Because this study was of a relatively short duration, the group provided input at two advisory group meetings. At the meetings there was discussion regarding the focus of the work, including a presentation on previous research on NHS111 services to provide a context for understanding the current work. The meetings also included the presentation and discussion of the findings of the review in order to explore key messages for patients that could inform the dissemination of the findings.

The discussion during one meeting was structured using a SWOT (strengths, weaknesses, opportunities and threats) analysis approach that revealed a number of potential concerns among patients as well as potential perceived benefits. The group members expressed some concern over the reliability and consistency of symptom checker algorithms, particularly if different systems are being used in different parts of the country. The economic benefits of the systems were also questioned in view of the high costs of programming and system development. Possible equity issues were identified because digital 111 might be less accessible to some groups (people with cognitive impairment were mentioned). The group expressed uncertainty about the impact of this type of service on the wider urgent and emergency care system (e.g. would the creation of a new access point complicate patient pathways and increase demand overall?). The possible vulnerability to external threats leading to the breakdown of the system or the loss of data was identified as a weakness. There was discussion of whether or not patients would feel able to trust the advice they were given. Some group members saw the desire of patients to be able to talk to someone about their problems for reassurance and empathy as a threat to the success of digital 111. Potential benefits included increased access to urgent-care advice at any time and appeal to younger people and to those who might feel anxious or embarrassed about discussing their problem with a health professional.

Involvement of the advisory group was also beneficial in highlighting some issues that had emerged from the systematic review and enabled the reviewers to structure the review findings while taking this into account. The group’s uncertainty about the probable impact of digital 111 was reflected in the report’s findings and recommendations for ongoing evaluation and further research. The review report also reflects the group’s relatively cautious attitude (while recognising the need to update the way services are accessed), which contrasts with the strong belief in some quarters that digital 111 will help to ensure that patients receive appropriate care more quickly while reducing ‘inappropriate’ visits to EDs and GP appointments. Advisory group members reported that they had limited experience of using symptom checkers for real-life health problems in the urgent-care setting, which echoes the limited available information and uncertainty regarding the probable effects of digital 111 services.

Study registration and outputs

The protocol was registered prospectively with PROSPERO (registration number CRD42018093564) and is also available via the National Institute for Health Research (NIHR) HSDR programme website (www.journalslibrary.nihr.ac.uk/programmes/hsdr/164717/) and the Sheffield HSDR Evidence Synthesis Centre website (https://scharr.dept.shef.ac.uk/hsdr/).