Review results

Some parts of this chapter have been reproduced from Chambers et al.1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The text includes minor additions and formatting changes to the original text.

Results of literature search

This chapter presents the studies that were included in the review. A PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-analyses) flow diagram (Figure 1) details the search process.

FIGURE 1 The PRISMA flow diagram. Some parts of this chapter have been reproduced from Chambers et al.1 © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY. Published by BMJ. This is an open access article distributed in accordance with the Creative Commons Attribution 4.0 Unported (CC BY 4.0) license, which permits others to copy, redistribute, remix, transform and build upon this work for any purpose, provided the original work is properly cited, a link to the licence is given, and indication of whether changes were made. See: https://creativecommons.org/licenses/by/4.0/. The figure includes minor additions and formatting changes to the original text. (Figure not reproduced here; see the original article.)

All titles and abstracts were screened by one researcher from the review team, with a subset (about 10%) of the titles and abstracts screened by one other researcher from the review team. A calculation of the inter-rater agreement was made. A kappa coefficient was calculated, demonstrating moderate agreement between reviewers [κ = 0.582, 95% confidence interval (CI) 0.274 to 0.889]. Any queries were resolved by discussion. A similar process was followed for final decisions on inclusion/exclusion, based on full-text documents.

Characteristics of included studies

We included 29 publications that represented 27 studies (one study13 had two associated comments14,15). Nine studies were performed in the UK. Seventeen studies (Table 1) evaluated symptom checkers as a self-contained intervention, of which eight covered a limited range of symptoms (e.g. respiratory16,17,25 or gastrointestinal28,30 symptoms that we considered to be ‘urgent’). The remaining studies in this group evaluated symptom checkers that covered a wider range of common urgent-care symptoms. Studies evaluated either a single system21,22,24,27 or multiple systems.6,13 We found only one study of a symptom checker specifically intended for the assessment of children’s symptoms, a development of the Strategy for Off-Site Rapid Triage (SORT) system for influenza-like illnesses.23 Two reports with some overlap of content evaluated the ‘babylon check’ application (app).26,27

Five studies7,31–34 evaluated symptom checkers as part of a broader self-assessment and consultation system (often referred to as electronic consultation or e-consultation). Study characteristics are summarised in Table 2. In this type of system, the role of symptom checkers is to help patients decide whether their symptoms require a consultation with a doctor or other health professional or can be dealt with by self-care. If a consultation is required, details of the symptoms and a request for an appointment or a call-back can be submitted electronically. This type of study is important because it considers the service in the broader context of the urgent and emergency care system. A limitation is that some studies focused mainly on the ‘downstream’ elements of the pathway (e.g. consultation with GPs) and provided limited data on the symptom checker element of the system.

A final group of five studies (Table 3) examined patient and/or public attitudes to online self-diagnosis in the context of urgent care.35–39

The included studies used a wide range of designs, some of which were challenging to classify. Studies that assessed the systems’ performance in terms of clinical and service use outcomes, including patient satisfaction, were generally observational or qualitative, although one web-based system was evaluated in two RCTs.17,25 Diagnostic accuracy was assessed by measuring performance in simulations (e.g. using vignettes to describe symptoms for a known condition) or by comparing the system’s performance with that of doctors in diagnosing the cause of symptoms and choosing an appropriate level of triage. This was carried out using either simulated data or real-world data (e.g. by asking patients to complete a symptom checker assessment before seeing a doctor and comparing the two assessments). Risk of bias in studies and the overall strength of evidence for different outcomes are discussed further below (see Risk-of-bias assessment and Overall strength of evidence assessment, respectively).

The publication status of the included studies also varied. In addition to peer-reviewed journal articles and reports, we included four studies published as conference abstracts18,28–30 and three reports characterised as grey literature. Two of these reports27,34 were written and published by the developers of specific systems and may therefore be subject to potential conflicts of interest. The third grey literature report was a draft report of NHS England’s ongoing pilot studies of NHS111 online in four regions of England.7 This report, dated December 2017, was not officially published by NHS England but was easily available online.

Characteristics of included systems

Using studies included in the review, we were able to extract data on eight systems using the TIDieST checklist. When appropriate, data from multiple studies were combined in one checklist. We noted that some of these systems were no longer in use. We were also unable to obtain sufficient data for some systems that are currently being used and/or evaluated. A summary of extracted data is presented below (Tables 4 and 5) and full data extractions can be found in Appendix 3.

Four of the included systems were designed to cover a full range of symptoms (see Table 4) and four covered a more limited range (three for influenza-like illness and one for minor respiratory symptoms; see Table 5). Most systems were accessed through web pages, often linked to health-care providers or government organisations. The ‘babylon check’ system was the main exception, being designed for access using a smartphone app. Published research studies provided relatively little detail about the systems that, in some cases, possibly reflected a need for commercial confidentiality. Details of published studies are included in Tables 4 and 5 for reference; the findings of these studies are summarised in the following section for each key outcome.

Results by outcome

Safety

Patient safety is an important outcome in evaluating any urgent-care intervention. Misdiagnosis or undertriage of serious conditions could lead to death or serious adverse events. However, none of the six included studies that reported on safety outcomes identified any problems or differences in outcomes between symptom checkers and health professionals (Table 6). Most of the studies compared system performance with that of health professionals using real or simulated data. The only study with no comparison group was the 6-month pilot study of webGP,34 which reported ‘no major incidents’.

Limitations of the studies included not being based on real patient data,27 covering only a limited range of conditions,16,23 and sampling a young and healthy population (students) that was not representative of the general population of users of the urgent-care system.22 Studies of e-consultation systems did not generally collect data on those respondents who decided not to seek an appointment, which limited their ability to assess any impact on safety for this group. Overall, the evidence should be interpreted cautiously as indicating no evidence of a detrimental impact on safety rather than evidence of no detrimental effect.

Clinical effectiveness

Only two studies reported on clinical effectiveness outcomes (Table 7), making it difficult to draw any firm conclusions. In the study by Little et al.,17 those who used the Internet Doctor website experienced longer illness duration and more days of illness that was rated as moderately bad or worse than the usual-care group, although the difference was not statistically significant. The online intervention in this study was designed to offer self-management advice for respiratory infections only. The pilot study of the webGP system34 reported that several patients received advice to seek treatment for serious symptoms that might otherwise have been ignored. However, no details or quantitative data were provided.

Costs and cost-effectiveness

Three included studies provided limited data on possible cost savings (Table 8). Based on 6 months of pilot data, Madan34 estimated savings of £11,000 annually for an average general practice (6500 patients) compared with current practice. The report also suggested that there might be a saving to commissioners equivalent to £414,000 annually for a Clinical Commissioning Group (CCG) covering 250,000 patients. These savings were specifically related to the self-reported diversion of patients from GP appointments to self-care and from urgent care to e-consultation and, as such, were associated with the symptom checker part of the system. Using similar methodology, the manufacturers of the ‘babylon check’ app claimed average savings of > £10 per triage compared with NHS111 by telephone, based on a higher proportion of patients being recommended to self-care.26 Neither study represents a formal cost-effectiveness analysis. The fact that these studies were produced by system manufacturers should be taken into account when interpreting their findings.

The other study reporting on costs32 concentrated on potential savings to practices from using e-consultation and found that savings depended on the percentage of face-to-face appointments avoided by use of e-consultation. Although important for the evaluation of e-consultation systems, this study has less relevance to the role of symptom checkers because patients completing an e-consultation have already decided that they need further contact with the health-care system.

Further economic evaluations should fully compare clearly defined alternatives and consider costs and benefits across the whole health-care system, particularly whether or not savings represent genuine cost reductions or merely transfers of costs from one part of the system to another.

Diagnostic accuracy

Eight studies reported at least some data on the diagnostic accuracy of symptom checkers (Table 9). One study18 was excluded from the table because although it contained some data on diagnostic accuracy of Google and WebMD symptom checkers, it was mainly concerned with patients’ ability to self-diagnose using the different systems. Most of the other studies attempted to compare the systems’ performance with that of health professionals using real patient data or simulations. Four of these studies23,28–30 evaluated systems designed to diagnose and triage specific symptoms or conditions and only three6,13,24 evaluated one or several ‘general purpose’ symptom checkers.

In spite of the diverse methods and comparisons in the included studies, almost all studies agreed that the diagnostic accuracy of symptom checkers was poor in absolute terms (e.g. in evaluating ‘vignettes’ designed to test knowledge of specific conditions when the correct diagnosis was already known by definition) or relative to that of health professionals. In the most comprehensive evaluation, Semigran et al.6 evaluated 23 symptom checkers across 770 standardised patient evaluations. Overall, the correct diagnosis was made in 34% of cases (95% CI 31% to 37%), although performance varied widely between symptom checkers, high- and low-acuity conditions, and common and rare conditions. When the same authors compared the 23 symptom checkers with physicians using 43 vignettes, physicians were more likely to list the correct diagnosis first (out of three differential diagnoses) (72.1% vs. 34%; p < 0.001) as well as among the top-three diagnoses (84.3% vs. 51.2%; p < 0.001).13

The only exception to the rule was an evaluation carried out at a student health centre.24 Using data from 59 participants who used the 24/7 WebMed system and who were subsequently treated at the health centre, the study found good agreement between chief complaint, 24/7 WebMed classification and provider diagnosis (κ-values of 0.89 to 0.94). This study differed from the others in using data from students rather than a general population sample. Data were obtained only from people who were actually treated, raising a potential issue of partial verification bias. In addition, the students’ complaints were generally common and uncomplicated, a scenario in which symptom checkers performed relatively well in the study by Semigran et al.13

Accuracy of disposition (triage and signposting to appropriate services)

The ability to distinguish conditions of different degrees of urgency and advise patients on an appropriate course of action is a vital component of symptom checkers and e-consultation services and is closely linked to both safety and service use. Six included studies6,7,21,22,26,27 reported on this outcome, and one30 evaluated a ‘general purpose’ symptom checker (Table 10). As with diagnostic accuracy, diverse methodologies and outcome measures were used.

The results overall presented a mixed picture, but most studies indicated that symptom checkers were inferior and/or more cautious in their triage advice than doctors or other health professionals. In their review of 23 symptom checkers, Semigran et al.6 found that the systems provided appropriate triage advice in 57% (95% CI 52% to 61%) of cases. The rate of appropriate triage advice was higher for emergency cases (80%) than non-emergency (55%) or self-care (33%) cases. Performance also varied across the systems evaluated, with correct triage ranging from 33% to 78%. Similarly, the NHS England pilot evaluation7 of four systems found that agreement with clinical experts varied from 30% to 95%, although the number of responses also varied, reducing the comparability of the results.

For abdominal pain, Berry et al.30 evaluated three symptom checkers and found that 33% of diagnoses were at the same level of urgency as physician diagnoses (emergency, non-emergency or self-care), 39% of cases were diagnosed as more serious and 30% of cases were diagnosed as less serious than the physician’s judgement. A similar level of agreement between algorithm and clinician (39%) was reported by Poote et al.,22 whereas the system evaluated by Nijland et al.21 advised patients to visit a doctor in 85% of cases, even when the symptoms were appropriate for self-care.

The only studies to report clearly equal or superior accuracy of disposition using an automated system were the evaluations of ‘babylon check’ by the company that developed the system. Middleton et al.27 reported that, when using patient vignettes, the app gave an accurate triage outcome in 88.2% of cases, compared with an accurate triage outcome in 75.5% of cases for doctors and in 73.5% of cases for nurses. When vignettes were delivered by a medical professional rather than actors, the accuracy of ‘babylon check’ increased to > 90%. A later report looked at triage results obtained as part of the NHS England pilot evaluation, concluding that all of the 74 referrals to urgent or emergency care were appropriate.26 However, this evidence should be treated with some caution as it was published without peer review and all of the authors were affiliated with the developer of the product.

Impact on service use/diversion

The widespread availability of online and digital systems for self-diagnosis and triage is intended to support self-management when appropriate and optimise the use of the urgent and emergency care system. However, unintended impacts are possible and for this reason it is important to monitor effects on service use, particularly the diversion of patients from urgent-care settings to primary care or self-care. Eight studies reported on this outcome (Table 11), although one16 of them merely stated that it was not possible to assess the effect of the intervention (a web-based influenza triage system) on patients’ use of health services.

Three of the studies reporting on service use evaluated e-consultation systems rather than symptom checkers as a single intervention. However, the evaluations of webGP/eConsult by Carter et al.31 and Cowie et al.32 reported on the use of the e-consultation part of the system rather than the symptom checkers. By contrast, the pilot evaluation of the system by its developers reported that 18% of users planned to book an appointment but chose not to do so.34 In addition, 14% of users reported that they would have attended a walk-in centre or other urgent-care service if they had not had access to the webGP system. These are promising findings but few details of the methodology are available from the report, for example the sample size of the survey and whether respondents were representative of webGP users or the general population of participating practices.

Further evidence on service use comes from the informally published draft report7 of the NHS England pilot evaluation of four online/digital systems in different regions of England. This report compared the recommendations of the digital systems with those of the NHS111 telephone service over a similar time period (the first months of 2017). Compared with the telephone service, the online and digital systems directed a slightly higher proportion of patients to self-care (18% vs. 14%) and a lower proportion of patients to other primary care services, such as general practices, dental services and pharmacists (40% vs. 60%). Again, these findings should be interpreted cautiously as they are based on short-term data and do not take account of differences between the different systems and pilot regions or between online/digital and telephone service users. The manufacturer’s data on the ‘babylon check’ app collected as part of the NHS England evaluation26 indicated that patients were more likely to be triaged to self-care by the app than by the NHS111 telephone service (40% vs. 14%). This figure includes people who received information leaflets on self-care as well as those who were actively triaged. If the former group is excluded, the figures for the two services are similar (14% of patients triaged by NHS111 vs. 15.6% of patients triaged by ‘babylon check’).

In their study of self-assessment for students attending a university health centre, Poote et al.22 found that the prototype system that they studied was able to identify a proportion of cases that doctors considered appropriate for self-care, suggesting a potential to reduce service use. Similarly, Little et al.’s17 RCT of a web-based symptom checker designed to support self-care for respiratory symptoms reported that patients in the intervention group had fewer contacts with doctors than patients in the usual-care control group, despite having a longer duration of illness and more days with relatively severe symptoms. This was balanced by an increase in contacts with the NHS Direct telephone service, and it should be noted that the system under evaluation recommended that people needing treatment should contact NHS Direct rather than go directly to a doctor. Finally, a study of young adults (students)38 found that the intention to seek treatment for a hypothetical illness was stronger when the diagnosis was made with the aid of WebMD or Google search than with no electronic aid.

In summary, there is some evidence that access to digital/online symptom checkers may influence patients’ use of health services and possibly increase self-care. However, most included studies reported surrogate indicators of service use that should be regarded as indirect evidence. The strongest evidence relates to a theory-based intervention designed to support self-care for a limited range of symptoms.17,25 Evidence for systems covering a broader range of symptoms is much weaker and is derived mainly from informally published non-peer-reviewed sources.

Patient compliance with triage advice

The effectiveness of digital and online symptom checkers will be influenced by both the quality of the triage advice provided and whether or not patients follow that advice. In particular, if patients choose to access further advice or seek appointments not recommended by the system, then the effect could be to increase overall demand on the urgent-care system or the wider health-care system.

Only two of the included studies reported specifically on patients’ compliance (or intention to comply) with the advice received (Table 12). The NHS England pilot evaluation7 in four regions asked participants in two of those regions (Suffolk and London) what they intended to do based on the advice received. It appears that the question was asked when patients were aware of the advice from the system, but it was unclear whether the evaluation covered real or hypothetical cases. No quantitative data were provided but the report stated that in the Suffolk pilot ‘overall users would have followed the advice given’7 (contains public sector information licensed under the Open Government Licence v3.0 www.nationalarchives.gov.uk/doc/open-government-licence/version/3/). However, those who were recommended to call 999 or attend an ED were more likely to seek advice from primary care or self-management. Similarly, in the London region there was generally good agreement between advice and intended action but patients who were recommended to call 999 or to go to an ED indicated that they would seek advice from a GP. These findings require further investigation, as delay in seeking treatment for serious conditions could be dangerous for the patient and ultimately costly for the health service. The study is also limited in that it asked about the patients’ intention to comply rather than their actual compliance.

In a study of a web-based triage system in the Netherlands, 192 patients were asked about their intention to comply immediately after receiving advice from the system.21 Thirty-five patients responded to a follow-up survey on actual compliance, of whom 20 (57%) reported that they had followed the advice. Compliance was correlated with intention to comply, which in turn was correlated with the patient’s attitude towards the advice received.

The available evidence on patients’ willingness to follow digital or online triage advice is thus limited in both quantity and quality. Given the importance of the outcome, this should be a priority for further research and evaluation.

Patient/carer satisfaction

Ten out of the 27 included studies had an outcome of patient/carer satisfaction. Five of these studies were investigating symptom checkers as a self-contained intervention.18,20,21,23,25 Four of these studies were evaluating symptom checkers as part of a broader e-consultation system.7,31,32,34 One study was examining the attitudes of the public to online self-diagnosis within the context of urgent care.37

One study reported on the general acceptability of the WebGP website in terms of timeliness, quality and experience for patients. However, general practice staff reported that online consultations interacted negatively with their own systems.31 Satisfaction with e-consultations was found to be high among respondents (91.4%), although only 6.5% of users responded to the survey.32 Similar high satisfaction rates were reported by Madan,34 although response rates were not reported. NHS England piloted a range of e-consultation systems with high rates of satisfaction for each.7 Sample sizes varied from small (17 respondents in Leeds) to large (1463 respondents in London). A number of different measures were used in the pilot studies. In Leeds and Suffolk, 75–80% of respondents were likely or extremely likely to recommend the service to friends and family, but sample sizes were small (17 in Leeds and 25 in Suffolk). Users in London rated the service on a scale of 1* to 5*, with 68% of users rating the service 4* or 5*.

Two of the studies investigating patient satisfaction considered self-triage tools for influenza.20,23 One of the studies was investigating the use of SORT for Kids23 and the other considered satisfaction with a practice website or telephone hotline designed to support self-management.20 In total, 90% of parents using SORT for Kids found the website ‘very easy’ to understand and 91.4% of parents found it ‘very easy’ to use.23 Qualitative feedback on patients’ satisfaction with a practice website or telephone hotline to support self-management of influenza demonstrated a high level of overall satisfaction.20 Additionally, findings indicated the ease of use, problem-free access and navigation, and the useful and adequate information that was used in a variety of ways by patients.

A study investigating satisfaction with the app Internet Doctor for minor respiratory problems found that overall satisfaction was higher in the Internet Doctor group than in the control (self-care) group (p = 0.002).25 However, there was a high attrition rate for the follow-up after 48 hours (from 714 users to 332 users) and the sample was generated from a population of undergraduate university students, so findings may not be representative of a general population.

A study investigating the experience of older adults searching for online health information18 found that they regularly commented on the use and navigation of the tools, the layout and features of the tools, and their difficulties with the computer programs. Findings indicate the importance of ease of navigating and troubleshooting symptom checkers to this population, as well as a general lack of confidence in self-diagnosis.

One of the studies21 investigating a symptom checker as a stand-alone intervention considered patient use and compliance with the medical advice provided and found that patient attitudes towards the advice received were neutral to positive.

Importance of ease of use of the application, navigation and troubleshooting

One included study37 surveyed two groups of users about their perceptions of symptom checkers. They measured expected usefulness, expected ease of use, trust in service provider, attitude towards using self-diagnosis and intention to use. The study found that there was a positive relationship between expected usefulness and attitude towards using self-diagnosis, as well as between expected ease of use and attitude towards using self-diagnosis, expected usefulness and ease of use, and expected usefulness/ease of use and trust in service provider. A relationship between behavioural intention and expected usefulness was not supported. These findings suggest that trust in the provider of the app is fundamental to the app’s perceived usefulness and ease of use and, indirectly, the users’ later behavioural intention.

Equity and inclusion

The results for the comparison of users versus non-users will be considered with the outcome equity and inclusion because there is significant overlap in the results of these outcomes. Fourteen out of the 26 included studies investigated the outcome of equity and inclusion or compared users and non-users. Seven of these studies were evaluating symptom checkers as a self-contained intervention.13,17,19–23 Five of these studies were evaluating an e-consultation system that included a symptom checker7,31–34 and two of these studies were investigating public attitudes towards symptom checkers.35,37

One study17 reported that patients who were classified as less socioeconomically deprived were more likely to agree to use Internet Doctor than decline participation, although no relationship was found between socioeconomic deprivation and results in this study or between e-consultation use and socioeconomic deprivation in another study.32

The association between e-consultation use and education levels was explored in one study.33 Patients with low to medium levels of education tended to be motivated towards indirect e-consultation (which involves contact with a health professional via e-mail), mainly to reduce uncertainty.

Evidence from included studies suggests that the users of e-consultation were more likely to be young (age range varied between studies),7,31,32,34 employed21,31,34 and female7,21,32,34 than non-users. One study also found a significantly larger use of e-consultation by white patients (78%) than patients of other ethnicities.32

A study investigating the use of health information or advice before ED or primary care centres in Sweden compared the characteristics of users with non-users.35 Only 10% of participants reported ever having consulted internet sources for information or advice on an urgent issue. More ED (49%) than primary care (12%) attendees were reported to have sought health information prior to their visit, although this information was not necessarily from an online source

Patient and public attitudes to online self-diagnosis

Seven cross-sectional papers were included in the review.18,19,35–39 Study characteristics are summarised in Table 13. The studies were carried out in the USA,18,36,38,39 Norway19,37 and Sweden35 and all but one, a conference abstract,38 were published articles.

Included studies used a range of cross-sectional research methods, including a review of online resources, surveys, interviews and ‘think aloud’ sessions to assess the quality, usability, acceptability of and the intention to use internet-based health-care information. Given the generally small samples of specific populations (e.g. students, older people, affluent) the generalisation of findings is limited, but the issues identified could be useful when considering the development of a new service.

Six out of the seven papers were assessed for quality, given that the one conference abstract fell outside our criteria for appraisal.38 The remaining studies were varied in terms of methods and how the methods fit with the appraisal criteria and, therefore, they were not graded; instead, strengths and weaknesses were discussed. All six papers included a clear statement of the research question and four of them presented clear outcome measures, but confounding variables were less clearly dealt with. Only two studies included a participant response rate,19,35 and both of these indicated that the rate was > 50%, but only five studies recruited participants. Blinding of assessors was either not appropriate or not stated for all of the studies. Other weaknesses included inconsistency of reporting the findings and a general lack of enough detail to make a decision regarding the risk of bias.

Aims of the included studies can be distinguished between those that examine characteristics of the supply side (online sources) and those of the demand side (users and potential users), or both. This indicates an interest in:

• the development of online resources for health-care information or advice
• behaviour of users in relation to online health-care information seeking.

These two issues are dealt with separately in the sections below.

Development of resources

On the provider side, websites developed to provide information resources to the public have been reported to vary in quality and content.36 Developers can certify and comply with the Health on the Net (HON) code40 to increase the quality and trustworthiness of the website. Trustworthiness may also be increased by not allowing open editing of the information provided. The goal of the provider can have an impact on the website’s usefulness; therefore, stating the aim of the website would allow users to make a decision on how useful the website is for their specific purpose.

The process of navigating sites was reported as an issue that could increase frustration, for example if the pathway takes a long time to go through, the signposting is not clear and/or the entered information is not recognised as saved into the system.19 Users were reported to be more likely to rush the task if the process was taking a long time, which increased the likelihood of self-diagnostic inaccuracy.19 The ease of the task and the confidence in performing the task were associated with self-diagnostic accuracy; therefore, optimising usability appears to be an important aspect when designing a health information system. Younger participants were reported to better understand navigation and system issues and, therefore, attention to the needs of older users may be required.

User behaviour

It is important that users understand the meaning of the information that they are being given and the questions that they are being asked. Included studies19,36 identified two barriers to using the services: low readability scores and user-reported language that was too clinical for lay readers. Requests for symptom information could be ambiguous, particularly in relation to measures (e.g. ‘how much’, ‘how many’ and ‘what colour’). The use of visual aids or more flexible options could help users to better interpret these types of questions.

Previous health knowledge and experience was reported to influence the accuracy of a self-diagnosis online, with prior health experience having a negative effect and prior health knowledge having a positive effect. Luger and Suls38 suggest that website design should incorporate a process that avoids patient decision-making based on ‘pattern rule’, whereby cognitive expediency can overrule more thorough, hypothetico-deductive reasoning.

In terms of intention to use, seeking health information was reported to be more likely prior to a visit to accident and emergency (A&E) than to primary care, but the sources of information and advice were not limited to online websites.35 Men were more likely to attend A&E without prior information, and younger people were just as likely to express the intention to seek help from a health professional for a minor ailment as a serious one. This suggests that, in attempting to avoid unnecessary attendances, it may be prudent to consider how to raise awareness of online health information in different populations. Online use of health information was reported to be more frequent for adult symptom checks than for children, and for chronic rather than acute symptoms, highlighting how users perceive telephone and internet triage systems differently.39 For example, telephone triage may provide the needed reassurance from a health professional, particularly for parents and in cases of acute symptoms, compared with visiting an online symptom checker.

Strengths and limitations

As well as small, non-diverse samples, this group of studies used a range of cross-sectional methods, some with comparators and some assessing particular systems. Expert assessment rather than lay assessment, as well as the use of scenario techniques for assessing the use of information systems, may lack external validity compared with alternatives. However, there are stated reasons for using these methods. None of the studies met all of the risk-of-bias criteria, mainly because of the methods used and a lack of detailed reporting. The review findings reflect the heterogeneity of the studies and they cannot be combined in anything but a narrative form. However, the study findings provide insight into potential issues that could be considered when designing an internet tool for health information and advice.

Risk-of-bias assessment

We assessed risk of bias in the two included RCTs17,25 using the Cochrane risk-of-bias tool.9 Thirteen studies16,18–21,24,31–33,35–37,39 were assessed with the tool for cross-sectional and cohort studies and four (six publications6,13–15,22,23) were assessed with the modified QUADAS tool.10 Seven grey literature reports and conference abstracts were not formally assessed for risk of bias.7,27–30,34,38 Identified limitations were extracted for all included studies, the risk-of-bias results are presented in Appendix 5.

No serious problems were identified with the two randomised trials, although blinding was somewhat unclearly reported. The studies evaluated using the cross-sectional/cohort study tool were variable both in design and risk of bias. Most studies addressed a clear research question in a defined population. Participation rates, when applicable, varied widely between studies. None of the studies provided a sample size justification, making their statistical power uncertain, and some findings were based on very small samples. Most of the studies had clearly defined outcome measures but none reported on blinding (e.g. of outcome assessment). Adjustment for confounders is important for observational studies but only three of the included studies were judged to have adjusted adequately for confounders (see Appendix 5).

The four ‘diagnostic’ studies assessed using the modified QUADAS tool had some important limitations. Two did not recruit representative samples of patients22,23 and the other two used patient vignettes, so were judged ‘unclear’ for this aspect.6,13 There were also issues with blinding and potential partial verification bias in individual studies.

With the possible exception of the two RCTs,17,25 the included studies generally had at least a moderate risk of bias. However, the diverse designs and objectives of the studies made risk of bias difficult to assess in some cases with the available tools. Grey literature reports containing relevant data were included in the review but not formally assessed for risk of bias. Reports prepared by individuals with a commercial interest in a specific system and published without independent peer review27,34 should be treated with particular caution because of possible conflicts of interest.

Overall strength of evidence assessment/evidence map

The overall strength of evidence for key outcomes was assessed using a variant of the scale used by Baxter et al.,12 which relates the evidence for particular outcomes to the size and quality of the whole body of evidence (Table 14). The rating scale was as follows: ‘stronger evidence’ represented generally consistent findings in multiple studies with a comparator group design or comparative diagnostic accuracy studies, ‘weaker evidence’ represented generally consistent findings in one study with a comparator group design and several non-comparator studies or multiple non-comparator studies, ‘very limited evidence’ represented an outcome reported by a single study and ‘inconsistent evidence’ represented an outcome where < 75% of the studies agreed on the direction of effect.

Using this scale, there is relatively strong evidence that digital and online symptom checkers have not been shown to present any major patient safety issues, that their diagnostic accuracy tends to be lower than that of health professionals and that patients who have used these systems generally show high levels of satisfaction (mainly in non-comparative studies). Areas where evidence is lacking or inconsistent include clinical effectiveness and cost-effectiveness, accuracy of disposition to appropriate services and patient compliance with advice received. The implications of these findings will be considered in Chapter 4.