Mobile technologies to support healthcare provider to healthcare provider communication and management of care

Abstract

Background

The widespread use of mobile technologies can potentially expand the use of telemedicine approaches to facilitate communication between healthcare providers, this might increase access to specialist advice and improve patient health outcomes.

Objectives

To assess the effects of mobile technologies versus usual care for supporting communication and consultations between healthcare providers on healthcare providers' performance, acceptability and satisfaction, healthcare use, patient health outcomes, acceptability and satisfaction, costs, and technical difficulties.

Search methods

We searched CENTRAL, MEDLINE, Embase and three other databases from 1 January 2000 to 22 July 2019. We searched clinical trials registries, checked references of relevant systematic reviews and included studies, and contacted topic experts.

Selection criteria

Randomised trials comparing mobile technologies to support healthcare provider to healthcare provider communication and consultations compared with usual care.

Data collection and analysis

We followed standard methodological procedures expected by Cochrane and EPOC. We used the GRADE approach to assess the certainty of the evidence.

Main results

We included 19 trials (5766 participants when reported), most were conducted in high‐income countries. The most frequently used mobile technology was a mobile phone, often accompanied by training if it was used to transfer digital images. Trials recruited participants with different conditions, and interventions varied in delivery, components, and frequency of contact. We judged most trials to have high risk of performance bias, and approximately half had a high risk of detection, attrition, and reporting biases. Two studies reported data on technical problems, reporting few difficulties. Mobile technologies used by primary care providers to consult with hospital specialists We assessed the certainty of evidence for this group of trials as moderate to low. Mobile technologies: ‐ probably make little or no difference to primary care providers following guidelines for people with chronic kidney disease (CKD; 1 trial, 47 general practices, 3004 participants); ‐ probably reduce the time between presentation and management of individuals with skin conditions, people with symptoms requiring an ultrasound, or being referred for an appointment with a specialist after attending primary care (4 trials, 656 participants); ‐ may reduce referrals and clinic visits among people with some skin conditions, and increase the likelihood of receiving retinopathy screening among people with diabetes, or an ultrasound in those referred with symptoms (9 trials, 4810 participants when reported); ‐ probably make little or no difference to patient‐reported quality of life and health‐related quality of life (2 trials, 622 participants) or to clinician‐assessed clinical recovery (2 trials, 769 participants) among individuals with skin conditions; ‐ may make little or no difference to healthcare provider (2 trials, 378 participants) or participant acceptability and satisfaction (4 trials, 972 participants) when primary care providers consult with dermatologists; ‐ may make little or no difference for total or expected costs per participant for adults with some skin conditions or CKD (6 trials, 5423 participants). Mobile technologies used by emergency physicians to consult with hospital specialists about people attending the emergency department We assessed the certainty of evidence for this group of trials as moderate. Mobile technologies: ‐ probably slightly reduce the consultation time between emergency physicians and hospital specialists (median difference −12 minutes, 95% CI −19 to −7; 1 trial, 345 participants); ‐ probably reduce participants’ length of stay in the emergency department by a few minutes (median difference −30 minutes, 95% CI −37 to −25; 1 trial, 345 participants). We did not identify trials that reported on providers' adherence, participants’ health status and well‐being, healthcare provider and participant acceptability and satisfaction, or costs. Mobile technologies used by community health workers or home‐care workers to consult with clinic staff We assessed the certainty of evidence for this group of trials as moderate to low. Mobile technologies: ‐ probably make little or no difference in the number of outpatient clinic and community nurse consultations for participants with diabetes or older individuals treated with home enteral nutrition (2 trials, 370 participants) or hospitalisation of older individuals treated with home enteral nutrition (1 trial, 188 participants); ‐ may lead to little or no difference in mortality among people living with HIV (RR 0.82, 95% CI 0.55 to 1.22) or diabetes (RR 0.94, 95% CI 0.28 to 3.12) (2 trials, 1152 participants); ‐ may make little or no difference to participants' disease activity or health‐related quality of life in participants with rheumatoid arthritis (1 trial, 85 participants); ‐ probably make little or no difference for participant acceptability and satisfaction for participants with diabetes and participants with rheumatoid arthritis (2 trials, 178 participants). We did not identify any trials that reported on providers' adherence, time between presentation and management, healthcare provider acceptability and satisfaction, or costs.

Authors' conclusions

Our confidence in the effect estimates is limited. Interventions including a mobile technology component to support healthcare provider to healthcare provider communication and management of care may reduce the time between presentation and management of the health condition when primary care providers or emergency physicians use them to consult with specialists, and may increase the likelihood of receiving a clinical examination among participants with diabetes and those who required an ultrasound. They may decrease the number of people attending primary care who are referred to secondary or tertiary care in some conditions, such as some skin conditions and CKD. There was little evidence of effects on participants' health status and well‐being, satisfaction, or costs.

Summary of findings

Background

Effective communication with other healthcare providers and access to specialist expertise is essential for increasing health services capacity and providing optimal care, especially in areas where there is a shortage of healthcare providers (AAP 2015). The widespread use of information and communication technologies (ICT) can potentially increase the capacity of health services by supporting communication between different providers, and providing rapid access to specialist expertise.

Description of the condition

By 2035 there will be a worldwide shortage of approximately 12.9 million skilled healthcare providers (Campbell 2013). The biggest gaps will occur in Southeast Asia and sub‐Saharan Africa, but elsewhere too this will be a problem due to larger ageing populations, the rising prevalence of non‐communicable diseases, migration patterns and high turnover of healthcare providers. Remote and rural areas, where populations are likely to be poorer, sicker and less educated, are particularly at risk (OPHI 2017; Wu 2016). Healthcare providers in those settings can be isolated and have limited interaction with colleagues and specialists, with few opportunities for mentoring, consultation with experts, or referrals to other healthcare providers.

Description of the intervention

Digital technologies are increasingly used to support health systems (WHO 2018) by providing flexible options for communication and the exchange of information. These technologies can be used for medical diagnostic, monitoring and therapeutic purposes, when participants are separated by distance or time or both, with the ultimate goal of improving the health of individuals and communities (Steinhubi 2013). Provision of health care at a distance is usually referred to as telemedicine (WHO 2018), and can be implemented through mobile or fixed devices.

The exchange of information can happen synchronously (when interactions happen in real time) or asynchronously (when there is a lag between the clinical information being transmitted and the response), and through different channels, including video‐conferencing, mobile applications, and secure messaging (Kruse 2017; WHO 2016). The use of mobile technologies can improve access to specialty care (Liddy 2019b), particularly for underserved communities (Källander 2013). Widespread mobile broadband connectivity means that even healthcare providers in remote areas can access and communicate with their peers, improving co‐operation (Aceto 2018). The World Health Organization (WHO) Global Observatory for eHealth conducted a survey of the WHO Member States on the use of eHealth (WHO 2016), and reported that of the 122 countries surveyed 70% reported on the use of mobile health devices for consultation between healthcare professionals. The most common areas were teleradiology, telepathology, and teledermatology (WHO 2016), with teleradiology programmes being widely used. Within this review our focus was on mobile technologies to support provider‐to‐provider communication and management of care.

In a bid to maximise the coverage of healthcare services and to decrease the cost of providing health care, governments and healthcare agencies in some countries have funded some type of telehealth programme for provision of care, including promoting communication and management of care between providers. Some examples include the Technology Enabled Care Services programme in England (NHS Commissioning Assembly 2015), the Scottish Centre for Telehealth and Telecare (SCTT 2017), the telehealth services provided within the Medicare programme in the USA (MedPAC 2016), the Asia eHealth Information Network (AeHIN 20017), the KwaZulu‐Natal Experience in South Africa (Mars 2012), and the Aga Khan Development Network Digital Health Programme, which covers remote communities in South‐Central Asia and East Africa (AKDN 2019).

How the intervention might work

The use of mobile technologies between healthcare providers for communication, consultations and patient management might contribute to developing professional skills and expertise, as well as optimising multidisciplinary communication (AAP 2015) and evidence‐based clinical practice. This is particularly relevant for settings where there is a shortage of healthcare providers, for instance in low‐ and middle‐income countries and in rural and remote areas (Källander 2013). By enabling healthcare providers who are geographically separated to exchange clinical information and knowledge, mobile technology can facilitate universal health coverage by increasing access to health care. In 2018 the WHO published a classification of digital health interventions to categorise the functionality of the different applications; using this classification as a guide we include interventions that are portable and facilitate remote healthcare provider communication or co‐ordination of referrals, or both (WHO 2018).

Despite the possibilities, telehealth applications have been inconsistently implemented, with varying degrees of success due to technological challenges, legal considerations, human and cultural factors, and uncertainty around economic benefits and cost effectiveness (WHO 2016), although this is changing. Overcoming these barriers requires evidence‐based implementation of guidelines, driven both by governmental and professional medical organisations; legislation on confidentiality, privacy and liability; and the involvement of stakeholders in designing, implementing and evaluating telemedicine applications, focusing on the safety and the effectiveness of applications (Agboola 2016).

Why it is important to do this review

The rapid progress of information and communication technologies is accelerating the evolution of remote communication between providers for the management of care. This review is one of a suite of 11 Cochrane Reviews that contributed to the WHO guideline on digital interventions for health systems strengthening (WHO 2019), and focuses on the effectiveness of mobile technologies for communication and management of care between healthcare providers who are in different locations. The effectiveness of mobile technologies to support patient‐to‐healthcare provider communication is being assessed in another review (Gonçalves‐Bradley 2018a). The rationale for conducting this review is to assess the effectiveness of mobile health technologies as a method for healthcare providers to communicate, diagnose and manage patients; and to assess acceptability, satisfaction, resource use and technical difficulties. Research into the latter has been particularly neglected (Coiera 2016), and can provide crucial information for successful implementation.

Objectives

To assess the effects of mobile technologies versus usual care for supporting communication and consultations between healthcare providers on healthcare providers' performance, acceptability and satisfaction, healthcare use, patient health outcomes, acceptability and satisfaction, costs, and technical difficulties.

Methods

Criteria for considering studies for this review

Types of studies

We include randomised trials reported as full‐text studies, conference abstracts and unpublished data, irrespective of their publication status and language of publication.

Types of participants

All types of healthcare providers (i.e. professionals, healthcare assistants, and lay health workers) providing patient care through mobile technologies. We included trials targeting people with any condition, regardless of their location, setting, diagnoses, or demographic factors such as age.

Types of interventions

We include trials comparing health care delivered through a mobile device versus usual care. We defined 'usual care' by the setting in which the trial took place, including face‐to‐face exchanges and communication through other non‐digital channels. We include trials of healthcare providers who were geographically separated and used information and communication technologies. We have focused exclusively on the exchange of clinical information over wireless and mobile technologies, mobile phones of any kind (but not analogue land‐line telephones), tablets, personal digital assistants and smartphones, and when the healthcare provider enquiry received a response in real‐time or as immediate as clinically appropriate. Communication channels through a mobile device can include text messaging, video messaging, social media, voice calls, voice‐over Internet protocol (VoIP), and video‐conferencing, through software such as Skype, WhatsApp or Google Hangouts.

We include:

trials in which the healthcare provider used mobile technologies, such as telemedicine applications, to seek clinical guidance and support from other qualified healthcare providers in order to deliver direct patient care. This included co‐ordination of referrals and requests for expert opinion and diagnosis;trials in which the provider(s) seeking guidance was at a different location from the provider(s) offering guidance; andtrials in which the provider(s) seeking guidance transmitted clinical information using a mobile device and the provider(s) offering guidance responded on any device, including stationary devices.

We include trials of telemedicine interventions if they were portable/mobile. We include trials assessing unspecified types of communication devices for transmitting clinical information, so long as they were mobile, since trials often failed to report this detail.

We include all health issues and did not restrict the content of clinical health information exchanged. We include trials where the digital component of the intervention was delivered as part of a wider package if we judged it to be the core component of the intervention.

We excluded:

pilot and feasibility studies (pilot study defined as "a version of the main study that is run in miniature to test whether the components of the main study can all work together" and feasibility studies as "pieces of research done before a main study"; Arain 2010);trials that compared different technical specifications of telecommunication technologies (e.g. different communication channels, software, etc.);trials in which the use of telecommunications technology was not directly linked to patient care;trials in which the primary purpose of the intervention was education/training;trials assessing the accuracy of a portable medical device.

Types of outcome measures

Main outcomes

Providers' adherence to recommended practice, guidelines or protocols.Time between presentation and management of the health condition.

Other outcomes

Healthcare use, including referrals, clinical examinations and hospitalisations.Participants' health status and well‐being, to include mortality and measures of health status such as the Nottingham Health Profile or the SF‐36 (McDowell 2006).Healthcare provider acceptability and satisfaction; this includes self‐reported acceptability and satisfaction, measured with a validated scale, such as the Physician Worklife Survey (Konrad 1999).Participant acceptability and satisfaction; this included self‐reported acceptability and satisfaction, measured with a validated scale, such as the Patient Satisfaction Scale (La Monica 1986).Costs, including cost to the user and cost to the service (e.g. human resources/time, training, supplies and equipment).Unintended consequences; these could include errors in interpreting the data; transmission of inaccurate data, loss of verbal and non‐verbal communication cues, issues of privacy and disclosure that might affect interpersonal relationships, negative impacts on equity, and technical difficulties, for example failure or delay in the message delivery.

Search methods for identification of studies

Electronic searches

An Information Specialist developed the search strategies in consultation with the review authors and WHO content experts. We used a minimum cut‐off search date of 2000, based on the increased availability and penetration of mobile devices from that date onwards (ITU 2019). Appendix 1 lists the search strategies and results. 
We searched the following databases until 22 July 2019:

Cochrane Central Register of Controlled Trials (CENTRAL; 2019, Issue 7), in the Cochrane Library;MEDLINE Ovid;Embase Ovid;POPLINE;WHO Global Health Library.

Searching other resources

Trial registries

We searched clinicaltrials.gov (clinicaltrials.gov) and the World Health Organization International Clinical Trials Registry Platform (who.int/ictrp).

Grey literature

We conducted a grey literature search in August 2017, to identify trials not indexed in the databases listed above. We searched for relevant systematic reviews and primary studies on similar topics using Epistemonikos (epistemonikos.org), a database of health evidence and health‐related systematic reviews. We searched the content in mHealthEvidence (mhealthevidence.org), a database of global literature on mHealth. We contacted authors of relevant trials/reviews to clarify reported published information and to seek unpublished results/data, as well as researchers with expertise relevant to the review topic. Moreover, WHO issued a call for papers through popular digital health communities of practice such as the Global Digital Health Network and Implementing Best Practices, to identify additional primary trials as well as grey literature. We performed a backward and forward search of the primary reference identified for each eligible trial.

Data collection and analysis

Selection of studies

We downloaded all titles and abstracts retrieved by electronic searching to reference management databases (Distiller and Covidence) and removed duplicates. For title and abstract screening, we used a machine‐learning classifier that is able to assign a probability score that a given record describes or does not describe a randomised trial (Wallace 2017). Two review authors (from AM, BB, DGB, GV, IRC, and NH) screened titles and abstracts of trials with at least a 10% probability of being a randomised trial, and one review author screened those with less than a 10% probability. We retrieved the full‐text trial reports/publication of all potentially eligible reports, and two review authors (from AM, BB, DGB, GV, IRC, and NH) screened the full text to identify trials for inclusion and to identify and record reasons for excluding the ineligible trials. We resolved any disagreement through discussion, and if required consulted a third review author (DGB or SS).

We listed trials that initially appeared to meet the inclusion criteria but that we later excluded in the Characteristics of excluded studies table. We collated multiple reports of the same trial so that each trial rather than each report was the unit of interest in the review. We also provided any information we could obtain about ongoing studies. We recorded the selection process in sufficient detail to complete a PRISMA flow diagram (Liberati 2009).

Data extraction and management

We used the EPOC standard data collection form and adapted it for trial characteristics and outcome data (EPOC 2017a); we piloted the form on five trials. One review author extracted the following characteristics and a second review author cross‐checked data (from AM, BB, DGB, GV, IRC, and NH).

Methods: trial design, unit of allocation, location and trial setting, withdrawals.Participants: number, mean age, age range, sex, inclusion criteria, exclusion criteria, dates conducted, other relevant characteristics.Interventions: function of the intervention (monitoring, consultation, therapy), intervention components (including type of technology and mode of delivery, frequency of data transmission), comparison, fidelity assessment. For this review, we defined monitoring as the continuous evaluation of the progress of symptoms or a condition over a period of time; consultation as an exchange between the healthcare provider and the participant, where the provider discusses the participant's health status and provides guidance, support, or information; and therapy as the ongoing management and care of a participant, to counteract a disease or disorder.Outcomes: main outcomes specified and collected, time points reported.Notes: funding for trial, ethical approval.

We contacted authors of included trials to seek missing data. We noted in the Characteristics of included studies table if outcome data were reported in an unusable way. We resolved disagreements by consensus or by involving a third review author (DGB or SS). We used Review Manager 5 (RevMan 5.3) for data management.

Assessment of risk of bias in included studies

One review author assessed risks of bias for each trial using the criteria outlined in the Cochrane Handbook for Systematic Reviews of Interventions (Higgins 2017), plus the guidance from the EPOC group (EPOC 2017b), and a second review author cross‐checked data (from AM, BB, DGB, GV, IRC, and NH). We resolved any disagreement by discussion or by involving a third review author (DGB or SS). We assessed the risks of bias according to the following domains.

Random sequence generation.Allocation concealment.Blinding of participants and personnel.Blinding of outcome assessment.Incomplete outcome data.Selective outcome reporting.Baseline outcomes measurement.Baseline characteristics.Other bias.

We judged the risk of each potential source of bias as being high, low or unclear, and provide a quotation from the trial report together with a justification for our judgement in the 'Risk of bias' table. We summarised the 'Risk of bias' judgements across different trials for each of the domains listed. We considered blinding separately for different key outcomes where necessary (e.g. for unblinded outcome assessment, risk of bias for all‐cause mortality may be very different than for a participant‐reported pain scale). We assessed incomplete outcome data separately for different outcomes. Where information on risk of bias relates to unpublished data or correspondence with a trialist, we noted this in the 'Risk of bias' table. We did not exclude trials on the grounds of their risk of bias but clearly reported the risk of bias when presenting the results of the trials.

When considering treatment effects, we took into account the risk of bias for the trials that contributed to that outcome.

We conducted the review according to the published protocol (Gonçalves‐Bradley 2018b) and reported any deviations from it in 'Differences between protocol and review'.

Measures of treatment effect

We estimated the effect of the intervention using risk ratios (RRs) and associated 95% confidence intervals (CIs) for dichotomous data. For continuous measures, we analysed the data based on the mean, standard deviation (SD) and number of people assessed to calculate the mean difference (MD) and 95% CI (Higgins 2019). We ensured that readers could interpret an increase in scores for continuous outcomes in the same way for each outcome, explained the direction of effect, and reported where the direction was reversed if this was necessary.

Unit of analysis issues

Six trials used a cluster design (Byamba 2015; Chang 2011; Eminović 2009; Iversen 2018; Piette 2017; Van Gelder 2017). Of those trials, all except one had controlled for unit‐of‐analysis errors by adjusting for clustering, and thus were not further re‐analysed.

We had planned to control for unit of analysis errors by re‐analysing the data after adjusting for clustering, using the intracluster correlation coefficient reported by the trials. When not reported, we calculated intracluster correlation coefficients estimates (Campbell 2000) and the formula 1+(M‐1)xICC, where M is the average cluster size (Higgins 2019). However, it was not possible to obtain average cluster size for Byamba 2015 and as such it is possible that there are potential unit of analysis errors associated with the effect estimates of that trial.

Dealing with missing data

We contacted investigators in order to verify key trial characteristics and obtain missing outcome data where possible (e.g. when a trial report was only available as an abstract). Whenever it was not possible to obtain data, we reported the level of missingness and considered how that might have impacted the certainty of the evidence.

Assessment of heterogeneity

We conducted meta‐analyses and calculated the I² statistic to measure heterogeneity among the trials in each analysis. We considered an I² value of 50% or more to represent substantial levels of heterogeneity, but this value was interpreted in light of the size and direction of effects and the strength of the evidence for heterogeneity, based on the P value from the Chi² test (Deeks 2017). We identified substantial heterogeneity for one of the outcomes (mortality), but were not able to explore it by prespecified subgroup analysis as there were not enough trials.

Assessment of reporting biases

We attempted to contact trial authors, asking them to provide missing outcome data. Where this was not possible, and we considered that the missing data might have introduced serious bias, we explored the impact of including such trials in the overall assessment of results. We were not able to explore possible publication bias through a funnel plot (Sterne 2011), as we did not combine a sufficient number of trials.

Data synthesis

We undertook meta‐analyses for outcomes when the interventions, participants, and underlying clinical question were similar enough for pooling to make sense (Borenstein 2009). As there was considerable heterogeneity, we applied a random‐effect model (Deeks 2017). A common way that trialists indicate the presence of skewed data is by reporting medians and interquartile ranges. When we encountered this we noted that the data were skewed and considered the implications.

'Summary of findings' table

Two review authors (DGB and MF) assessed the certainty of the evidence (high, moderate, low, and very low) using the five GRADE considerations: risk of bias, inconsistency, imprecision, indirectness, and publication bias) (Guyatt 2008). We used methods and recommendations described in the Cochrane Handbook for Systematic Reviews of Interventions (Schünemann 2017) and the EPOC worksheets (EPOC 2017c), using GRADEpro software (GRADEpro GDT). We resolved disagreements on certainty ratings by discussion and provided justification for decisions to down‐ or upgrade the ratings using footnotes in the table, making comments to aid readers' understanding of the review where necessary. We used plain language statements to report these findings in the review (EPOC 2017d).

We created 'Summary of findings' tables for the following outcomes in order to draw conclusions about the certainty of the evidence within the text of the review:

Providers' adherence to recommended practice, guidelines or protocols;Time between presentation and management of the health condition;Healthcare use;Participants' health status and well‐being;Participant and provider acceptability or satisfaction with the intervention;Costs;Technical problems.

We created three 'Summary of findings' tables, according to the setting where the intervention was delivered (primary, secondary and community care), as the populations in those settings, both healthcare providers and participants, are substantially different.

We considered whether there was any additional outcome information that we were not able to incorporate into meta‐analyses, noted this in the tables and stated whether it supports or contradicts the information from the meta‐analyses. When it was not possible to meta‐analyse the data, we summarised the results in the text and in the 'Comments' section of the 'Summary of findings' tables.

Subgroup analysis and investigation of heterogeneity

We categorised trials by setting (community, primary and secondary care), according to healthcare provider type, e.g. primary care doctors' or nurses' communication with hospital‐based specialists, or community health workers consulting with clinic staff.

We planned to use the following outcomes in subgroup analysis.

Time between presentation and management of the health condition.Participants' health status and well‐being.

We planned to use the formal statistical techniques of Mantel‐Haenszel and regression to test for subgroup interactions (Mantel 1959) but due to the limited number of studies we could not use this technique.

Sensitivity analysis

We planned to perform sensitivity analyses defined a priori to assess the robustness of our conclusions and explore the impact on effect sizes. This would have involved restricting the analysis to published trials and to trials at low risk of bias. We did not perform sensitivity analyses as there were no unpublished trials and within the pooled analyses all the trials had the same risk of bias for the relevant 'Risk of bias' criteria.

Results

Description of studies

We identified 19 published randomised trials of mobile technologies to support healthcare provider to healthcare provider communication and management of care (see Characteristics of included studies).

Results of the search

We retrieved 20,949 records for title and abstract screening, screened the full‐text of 2041 citations and included 19 trials (35 citations) (Armstrong 2018; Azogil‐López 2019; Byamba 2015; Chang 2011; Davis 2003; Eminović 2009; Gulacti 2017; Iversen 2018; Liddy 2019a; Mansberger 2015; Orlandoni 2016; Pak 2007; Piette 2017; Riordan 2015; Sutherland 2009; Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2002; Whited 2013). In addition, we identified 15 ongoing trials (ACTRN12617000389303; ACTRN12618001007224; Gervès‐Pinquié 2017; Jeandidier 2018; Källander 2015; Koch 2018; Nakayama 2016; Stevanovic 2017; NCT02821143; NCT02986256; NCT03137511; Done 2018; NCT03559712; NCT03662256; Xu 2017). A total of 441 records were eligible for the associated review on mobile technologies to support patient to healthcare provider communication and management of care (Gonçalves‐Bradley 2018a). Figure 1 presents the results of the search.

1 Flow diagram (Figure not reproduced here; see the original article.)

Included studies

Trial populations

Seventeen trials included 5766 participants, while two trials did not report the specific number of participants (Liddy 2019a; Riordan 2015). The number of healthcare professionals recruited ranged from one general practitioner (GP) consulting with one ophthalmologist (Davis 2003), to another trial that randomised 113 GPs consulting with several specialty physicians (Liddy 2019a). Most of the trials involved primary care professionals consulting with specialists, namely dermatologists (Armstrong 2018; Byamba 2015; Eminović 2009; Pak 2007; Piette 2017; Whited 2002; Whited 2013), ophthalmologists (Davis 2003; Mansberger 2015), nephrologists (Van Gelder 2017) or radiologists (Sutherland 2009). In two studies more than one type of specialist was involved (Azogil‐López 2019; Liddy 2019a). The GPs mainly worked in urban settings and consulted with specialists also located in urban settings (N = 11). In four studies the GPs were located in rural settings, and consulted with providers in urban settings. There was one trial each for community‐based peer health workers consulting with clinic staff (Chang 2011), home‐visiting nursing staff consulting with a hospital physician (Orlandoni 2016), rural‐based physical therapists consulting with rheumatologists (Taylor‐Gjevre 2018), and community nurses consulting with specialist nurses or podiatrists (Iversen 2018). Two trials reported on emergency physicians consulting with hospital‐based specialists (Gulacti 2017; Riordan 2015).

All trials recruited adults, with Sutherland 2009 also recruiting adolescents and Azogil‐López 2019 recruiting participants aged seven years and older, and Orlandoni 2016 specifically recruiting participants aged 65 years and older. Three trials recruited participants with diabetes (Davis 2003; Mansberger 2015; Iversen 2018), and one with rheumatoid arthritis (Taylor‐Gjevre 2018). Seven trials recruited participants with a range of conditions seeking referral to a dermatologist (Armstrong 2018; Byamba 2015; Eminović 2009; Pak 2007; Piette 2017; Whited 2002; Whited 2013), two trials recruited participants attending the emergency department (Gulacti 2017; Riordan 2015) or requiring a hospital referral (Azogil‐López 2019; Liddy 2019a), and one trial each recruited participants requiring a trans‐abdominal or trans‐vaginal ultrasound (Sutherland 2009) or with chronic kidney disease (Van Gelder 2017). The two remaining trials recruited participants receiving antiretroviral therapy (Chang 2011) and home enteral nutrition (Orlandoni 2016).

Setting

Trials were mainly conducted in North America (9 trials) and Europe (six trials), with one trial each conducted in the Dominican Republic, Turkey, and Uganda, and Mongolia.

Interventions

The trials included in the review evaluated interventions that varied in mode of delivery, number of sessions, and healthcare providers involved. All trials used a portable device, 10 of them using a portable device to obtain clinical images which were then transmitted for further assessment (Armstrong 2018; Byamba 2015; Davis 2003; Eminović 2009; Mansberger 2015; Pak 2007; Piette 2017; Sutherland 2009; Whited 2002; Whited 2013). Four trials used mobile phones for text messages and voice calls (Chang 2011), secure messaging (Gulacti 2017), audio‐conferencing system (Azogil‐López 2019), and for interactive web‐based record and voice calls (Iversen 2018). Two trials used a tablet for secure messaging (Riordan 2015) or video consultation (Orlandoni 2016), whereas one trial employed a laptop for video consultation (Taylor‐Gjevre 2018). The remaining trials used an electronic health record system for eConsults, which could also be implemented through mobile phones (Liddy 2019a; Van Gelder 2017).

The trials also varied in the frequency and duration of contacts between the healthcare providers, with most trials consisting of a single consultation (e.g. Eminović 2009).

Although the control group was always described as receiving usual care, the description of the specific care received varied. For trials conducted in primary care, 'usual care' generally consisted of a referral for a face‐to‐face appointment in secondary care (Byamba 2015; Eminović 2009; Liddy 2019a; Pak 2007; Whited 2002; Whited 2013) or a reminder to book an appointment (Davis 2003; Mansberger 2015; Piette 2017; Sutherland 2009). For one trial that used a social media platform for emergency department physicians to communicate with specialists within the same hospital (Gulacti 2017), 'usual care' was to consult by phone, sending all clinical information verbally. For trials conducted in the community, 'usual care' was typically face‐to‐face appointments with specialists, either at the participant's home (Orlandoni 2016) or at outpatient clinics (Iversen 2018; Taylor‐Gjevre 2018).

Several trials reported on additional components of the intervention (Table 4). Nine reported the delivery of training (Armstrong 2018; Byamba 2015; Chang 2011; Eminović 2009; Iversen 2018; Mansberger 2015; Piette 2017; Sutherland 2009; Taylor‐Gjevre 2018), which usually focused on how to acquire digital images or use the web‐based system. For one trial of eConsult, the specialists received financial incentives for each eConsult they undertook (Liddy 2019a), and two trials provided monetary incentives for participants to take part (Armstrong 2018) or to complete follow‐up assessment (Mansberger 2015). Two trials reported that participants whose healthcare providers were allocated to the intervention group had increased access to health care, either directly (Armstrong 2018) or indirectly (Chang 2011).

Funding, ethical approval, and conflict of interest

Sixteen trials reported funding sources, all of which were provided by medical research institutes or university funding bodies. One of the trials also received funding from a biopharmaceutical company (Van Gelder 2017). Three trials did not report ethical or institutional review board approval (Byamba 2015, letter; Davis 2003, short report; Riordan 2015, conference abstract).

For three trials one or more members of the author team reported financial support from pharmaceutical companies (Armstrong 2018, 3/29 authors; Van Gelder 2017 1/10 authors; Whited 2013, 1/18 authors). The lead author of Pak 2007 was the co‐founder of a web‐based consultation service identical to that used in the intervention. Six studies did not report conflicts of interest (Chang 2011; Davis 2003; Riordan 2015; Sutherland 2009; Taylor‐Gjevre 2018; Whited 2002), and for the remaining nine studies the authors had no known conflict of interest.

Excluded studies

We excluded 1544 full texts, of which we report on 22 excluded trials (See Characteristics of excluded studies). The most frequent reason for excluding trials was the explicit use of non‐mobile equipment (eight trials).

Risk of bias in included studies

Figure 2 presents a graph for risk of bias and Figure 3 summarises risk of bias.

2 Risk of bias graph: review authors' judgements about each risk of bias item presented as percentages across all included studies. (Figure not reproduced here; see the original article.)

3 Risk of bias summary: review authors' judgements about each risk of bias item for each included study. (Figure not reproduced here; see the original article.)

Allocation

Fourteen trials described the generation of the randomisation schedule, and were judged at low risk of bias (Armstrong 2018; Azogil‐López 2019; Byamba 2015; Eminović 2009; Gulacti 2017; Iversen 2018; Liddy 2019a; Mansberger 2015; Orlandoni 2016; Pak 2007; Piette 2017; Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2013), one trial that 'tossed a coin' was judged as high risk of bias (Sutherland 2009), and we rated the remaining trials at unclear risk of bias. Fifteen trials were judged at low risk of bias for allocation concealment (Armstrong 2018; Azogil‐López 2019; Byamba 2015; Chang 2011; Eminović 2009; Iversen 2018; Liddy 2019a; Mansberger 2015; Orlandoni 2016; Pak 2007; Piette 2017; Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2002; Whited 2013), one at high risk (Sutherland 2009), and the remaining trials were unclear due to a lack of information.

Eight trials reported baseline outcome measurements that were similar between groups, thus being assessed at low risk of bias (Armstrong 2018; Chang 2011; Eminović 2009; Liddy 2019a; Orlandoni 2016; Taylor‐Gjevre 2018; Whited 2002; Whited 2013), and the remaining 11 trials were assessed as being at unclear risk of bias. Ten trials reported similar baseline characteristics between groups and we judged them to be at low risk of bias (Armstrong 2018; Chang 2011; Eminović 2009; Gulacti 2017; Liddy 2019a; Mansberger 2015; Orlandoni 2016;Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2013), three trials reported differences between groups at baseline and we judged them to be at high risk of bias (Azogil‐López 2019; Pak 2007; Piette 2017), and the remaining six trials were unclear.

Blinding

Due to the nature of the intervention it was often not possible to blind participants or healthcare professionals. We judged 16 trials to be at high risk of performance bias, and three at unclear (Byamba 2015; Davis 2003; Riordan 2015).

For objective outcomes we assessed six trials to be at high risk of detection bias (Mansberger 2015; Orlandoni 2016; Piette 2017; Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2013), eight trials to be at low risk of bias and five trials to have an unclear risk of bias (Byamba 2015; Chang 2011; Davis 2003; Eminović 2009; Riordan 2015). For subjective outcomes we assessed eight trials to be at high risk of detection bias (Armstrong 2018; Eminović 2009; Iversen 2018; Piette 2017; Taylor‐Gjevre 2018; Van Gelder 2017; Whited 2002; Whited 2013), one trial to be at low risk of bias (Pak 2007), and two trials to have an unclear risk of bias (Davis 2003; Riordan 2015). Eight trials did not collect data on subjective outcomes.

Incomplete outcome data

Eight trials had high rates of incomplete outcome data and we judged them to be at high risk of attrition bias (Azogil‐López 2019; Chang 2011; Eminović 2009; Liddy 2019a; Orlandoni 2016; Pak 2007; Taylor‐Gjevre 2018; Whited 2013), and nine trials at low risk of attrition bias and were unclear about two trials (Davis 2003; Riordan 2015).

Selective reporting

We judged nine trials to be at high risk of reporting bias, as either outcomes were not reported per protocol (Armstrong 2018; Eminović 2009; Gulacti 2017; Iversen 2018; Taylor‐Gjevre 2018; Whited 2013) or publications were found for the same trial without cross‐reference (Chang 2011; Pak 2007; Whited 2002). For three trials it was not possible to make a judgement due to a lack of information (Davis 2003; Mansberger 2015; Riordan 2015), and seven trials had a low risk of reporting bias.

Other potential sources of bias

We judged other potential sources bias as unclear in three trials, two because there was not enough information (Davis 2003; Riordan 2015), and the other due to several methods being reported to collect outcome data due to problems with follow‐up (Eminović 2009). We judged one trial to have a high risk of other potential sources of bias, as data collection methods differed for the two trial groups and were not clearly reported (Mansberger 2015). There was no other apparent source of bias for the remaining trials and we judged them to be at a low risk of bias.

Effects of interventions

See: Table 1; Table 2; Table 3

Comparison 1: Mobile technologies used by primary care providers to consult with hospital based specialists

Thirteen trials reported on mobile technologies used by primary care providers to consult with hospital‐based specialists. The studies involved GPs consulting with dermatologists (Armstrong 2018; Byamba 2015; Eminović 2009; Pak 2007; Piette 2017; Whited 2002; Whited 2013), ophthalmologists (Davis 2003; Mansberger 2015), radiologists (Sutherland 2009), nephrologists (Van Gelder 2017), or different specialists (Azogil‐López 2019; Liddy 2019a). The mobile component of the interventions consisted of a non‐mydriatic camera for retinal screening (Davis 2003; Mansberger 2015), portable ultrasound (Sutherland 2009), teledermatology to send digital images (Armstrong 2018; Byamba 2015; Eminović 2009; Pak 2007; Piette 2017; Whited 2002; Whited 2013), and eConsult through audio‐conferencing or secure direct messaging between healthcare providers, with a mobile component (Azogil‐López 2019;Liddy 2019a; Van Gelder 2017). For an overview of the evidence please refer to Table 1.

Main outcomes

1. Providers' adherence to recommended practice, guidelines or protocols

One trial reported on the use of telenephrology by nephrologists to communicate with primary care providers for people with chronic kidney disease (CKD) (Van Gelder 2017). The authors found little or no difference for providers' adherence to the advised monitoring criteria from national CKD guidelines, as measured by monitoring of disease progression and metabolic parameters (3004 participants; moderate‐certainty evidence; Analysis 1.1). Follow‐up was not reported.

2. Time between presentation and management of the health condition

Four trials reported on time between presentation and management of the health condition (656 participants; moderate‐certainty evidence; Analysis 2.1). Two trials recruited GPs who collected digital images from people with a skin condition and consulted with hospital‐based dermatologists on how to interpret them, reporting that people received the required treatment from their dermatologist in less time than those allocated to the control group: for Whited 2002 mean difference −40.5 days, 95% CI −23 to −58 days (275 participants); Piette 2017 reported a median of 4 days for the intervention group (IG) and 40 days for the control group (CG), with an adjusted hazard ratio (HR) of 2.55, P = 0.01 (103 participants). A third trial recruited GPs who shared ultrasound images with radiologists, finding little or no difference between groups on median time to participant follow‐up or diagnosis (Sutherland 2009; 105 participants). Azogil‐López 2019 recruited GPs who either referred their participants to an in‐person hospital appointment (control group) or to an audio‐consultation (intervention group), finding that those allocated to the audio‐consultation waited for less time (median −27 days, 99% CI −20 to −33 days; 173 participants). Follow‐up, when provided, ranged between three and six months.

Other outcomes

1. Healthcare use

Nine trials reported on various forms of healthcare use, including referrals, screening examinations, outpatient visits and hospitalisations (4810 participants; moderate‐certainty evidence; Analysis 3.1).

Four trials recruited GPs who consulted with dermatologists through the use of digital images (Byamba 2015; Piette 2017; Whited 2002; Whited 2013; 4 trials, 1075 participants; follow‐up between three and nine months, when reported), finding that those participants allocated to the intervention group were less likely to subsequently receive a referral for an appointment with a dermatologist, visit a dermatology clinic, or be referred to tertiary care: risk ratio (RR) ranged from 0.28 (95% CI 0.21 to 0.38) to 0.82 (0.75 to 0.88). We did not retain the meta‐analysis because of high statistical heterogeneity (Analysis 3.2; I² = 91%).

One trial of eConsults between PCPs and nephrologists reported that there was little or no difference between groups for referral rate (odds ratio (OR) 0.61, 95% CI 0.31 to 1.23; Van Gelder 2017; 3004 participants). Another trial of eConsults between PCPs and a range of specialists also found little or no difference between groups for face‐to‐face referral (RR 0.93, 95% CI 0.85 to 1.03; Liddy 2019a).

Two trials of retinopathy screening for participants with diabetes (Davis 2003; Mansberger 2015) reported that those allocated to the intervention group were more likely to receive a screening examination (2 trials, 626 participants; 12 months follow‐up when reported). High statistical heterogeneity precluded retaining the meta‐analysis (Analysis 3.3; I² = 85%). Another trial of GPs consulting with radiologists about participants requiring a trans‐abdominal or trans‐vaginal ultrasound found that participants allocated to the intervention group were more likely to receive an ultrasound (RR 3.92, 95% CI 2.11 to 7.31; Sutherland 2009; 105 participants).

2. Participants' health status and well‐being

Two trials reported on a dermatologist providing feedback to GPs based on digital images, finding similar scores between those allocated to the intervention and the control group, for general health status at 12‐month follow‐up (Armstrong 2018), as well as quality of life and health‐related quality of life as reported by the participants, at nine‐month follow‐up (Whited 2013) (2 trials, 622 participants; moderate‐certainty evidence; Analysis 4.1). Two teledermatology trials reported on clinical course as assessed by dermatologists at four‐ (Pak 2007) and nine‐month follow‐up (Whited 2013), finding little or no difference between groups in clinical course (2 trials, 769 participants; moderate‐certainty evidence; Analysis 4.2).

3. Healthcare provider acceptability and satisfaction

Two trials (378 participants) recruited GPs who consulted with dermatologists using digital images (low‐certainty evidence); Piette 2017 reported little or no difference between groups for acceptability or satisfaction, and Whited 2002 reported that GPs allocated to the intervention were more likely to agree that participants received timely appointments and to be satisfied with the consult process than GPs allocated to the control group. One additional trial (Van Gelder 2017) reported on satisfaction for healthcare professionals allocated to the intervention group (Analysis 5.1).

4. Participant acceptability and satisfaction

Four trials (972 participants, low‐certainty evidence; Analysis 5.2) recruiting GPs who consulted with dermatologists through the use of digital images reported little or no difference in participant satisfaction between those allocated to the intervention or to care as usual (Eminović 2009; Piette 2017; Whited 2002; Whited 2013).

5. Costs

Six trials reported costs (5423 participants; low‐certainty evidence; Analysis 6.1). One teledermatology trial reported that the expected cost per participant per visit was higher for the intervention group (Whited 2002; 275 participants); a second teledermatology trial reported that the total direct costs were lower for the comparison group (Pak 2007; 698 participants; MD USD −4678, 95% CI −4720 to −4635), and that this difference was offset by the lost productivity for participants allocated to the control group (MD USD 14,409, 95% CI 14,398 to 14,419). Another teledermatology trial reported little or no difference between groups for total costs per participant from the healthcare perspective (MD USD 30, 95% CI USD −79 to 20), and from the societal perspective that included the cost of loss of productivity (MD USD −82, 95% CI −12 to −152) per participant allocated to the intervention (Whited 2013; 391 participants). Two trials (teledermatology and telenephrology, respectively) reported little or no difference between groups for costs (Eminović 2009, 605 participants; MD EUR 32.5, 95% CI −29.0 to 74.7; Van Gelder 2017; 3004 participants; IG: EUR 453.86, 95% CI 392.98 to 514.74; CG EUR 433.74, 95% CI 387.64 to 479.84, P = 0.60). One teledermatology trial set in rural areas in Mongolia reported lower costs associated with the intervention group, mainly explained by the long distances that those allocated to the control group had to travel, which was avoided with teledermatology (Byamba 2015; 450 participants, IG: USD 320, CG: 3174, difference USD 2854).

6. Unintended consequences

Four trials reported on the quality of the data transmitted (Analysis 7.1). However, only one trial recruiting GPs consulting with dermatologists about images they took from their participants reported data for both groups (Pak 2007), reporting that 10 images from each group were lost due to technical problems (1 trial, 698 participants; moderate‐certainty evidence). The remaining trials reported results for the intervention group only (Piette 2017, Sutherland 2009, Whited 2002).

One trial where GPs could consult with dermatologists about people with psoriasis collected data about mortality as part of adverse events, reporting one death for each group (IG: 1/148; CG: 1/148; Armstrong 2018).

Comparison 2: Mobile technologies for communication between specialists in the emergency department

Two trials reported on mobile technologies for communication between physicians and specialists in the emergency department (Gulacti 2017; Riordan 2015), using a smartphone application for secure messaging. For an overview of the evidence please refer to Table 2.

Main outcomes

1. Providers' adherence to recommended practice, guidelines or protocols

Neither of the trials of mobile technologies for communication between specialists in the emergency department reported data on providers' adherence.

2. Time between presentation and management of the health condition

One trial that recruited emergency physicians who consulted with specialist physicians using a smartphone application reported that participants allocated to the intervention group were probably either admitted to hospital or discharged in slightly less time from the emergency department (median difference −12 minutes, 95% CI −19 to −7; 345 participants; moderate‐certainty evidence) (Gulacti 2017; Analysis 8.1).

Other outcomes

1. Healthcare use

One trial reported that participants seen by emergency physicians allocated to the intervention group probably had a shorter length of emergency department stay (median difference −30 minutes, 95% CI −37 to −25 minutes; 345 participants; moderate‐certainty evidence; Analysis 9.1, Gulacti 2017).

2. Participants' health status and well‐being

Neither of the trials of mobile technologies for communication between specialists in the emergency department reported data on participants' health status and well‐being.

3. Healthcare provider acceptability and satisfaction

Neither of the trials on mobile technologies for communication between specialists in the emergency department reported data on healthcare provider acceptability or satisfaction.

4. Participant acceptability and satisfaction

Neither of the trials on mobile technologies for communication between specialists in the emergency department reported on participant acceptability and satisfaction.

5. Costs

Neither of the trials on mobile technologies for communication between specialists in the emergency department reported data on costs.

6. Unintended consequences

Gulacti 2017 reported that there were no technical problems during the course of the trial (Analysis 10.1).

Comparison 3: Mobile technologies used by community health or home‐care workers

Four trials reported on mobile technologies used by community‐based health workers or home‐care workers. The professionals involved were community‐based peer health workers consulting with clinic staff about receiving antiretroviral therapy (Chang 2011); community nurses consulting with diabetes specialist nurses or podiatrists about adults with Type 2 diabetes (Iversen 2018); home‐care nurses consulting with hospital specialists about home enteral nutrition (Orlandoni 2016); and rural‐based physical therapists consulting with urban‐based rheumatologists (Taylor‐Gjevre 2018). The mobile‐based component of the interventions consisted of mobile phone, teledermatology, video‐consultations, and interactive web‐based records, respectively. For an overview of the evidence please refer to Table 3.

Main outcomes

1. Providers' adherence to recommended practice, guidelines or protocols

None of the trials of mobile technologies used by community‐based health workers reported data on providers' adherence.

2. Time between presentation and management of the health condition

None of the trials of mobile technologies used by community health workers reported data on time between presentation and management of the health condition.

Other outcomes

1. Healthcare use

Two studies reported on outpatient clinic and community nurse consultations (370 participants, moderate‐certainty evidence). Iversen 2018 recruited community nurses consulting with diabetes specialist nurses and podiatrists about adults with new diabetes‐related foot ulcers, reporting little or no difference between groups for outpatient consultations (0.48 fewer consultations in the intervention group, 95% CI −1.46 to 0.49) or community nurse consultations (0.92 more consultations in the intervention group, 95% CI −0.70 to 2.53). One trial (188 participants) that recruited home‐visiting staff who consulted with hospital physicians through video‐conferencing about older adults treated with home enteral nutrition reported little or no difference for healthcare use, as measured by outpatient visits (Incidence rate ratio 95% CI 0.65 to 1.30, P = 0.62) and hospitalisations (Incidence rate ratio 95% CI 0.54 to 1.19, P = 0.26) (Orlandoni 2016; low‐certainty evidence; Analysis 11.1).

2. Participants' health status and well‐being

Two trials, one recruiting community‐based peer health workers consulting with clinic staff about adults who were receiving or started receiving antiretroviral therapy (Chang 2011) and another recruiting community nurses consulting with diabetes specialist nurses and podiatrists about adults with new diabetes‐related foot ulcers (Iversen 2018), reported mortality at 11‐ to 12‐month follow‐up (RR 0.82, 95% CI 0.55 to 1.22 and RR 0.94, 95% CI 0.28 to 3.12, respectively; 1157 participants; low‐certainty evidence; Analysis 12.1).

One trial (85 participants) of rural‐based physical therapists consulting with urban‐based rheumatologists about adults with a clinical diagnosis of rheumatoid arthritis reported little or no difference between groups for health‐related quality of life and disease activity (low‐certainty evidence) (Taylor‐Gjevre 2018; Analysis 12.1).

3. Healthcare provider acceptability and satisfaction

None of the trials of mobile technologies used by community health workers reported data on healthcare provider acceptability or satisfaction.

4. Participant acceptability and satisfaction

Two trials (178 participants) reported on participants' experience with healthcare (Iversen 2018) and satisfaction with the intervention (Taylor‐Gjevre 2018), reporting little or no difference between those allocated to the intervention or the control groups (moderate‐certainty evidence) (Analysis 13.1).

5. Costs

One trial reported the total cost of running the intervention and cost per participant for the intervention group only (Chang 2011, Analysis 14.1).

6. Unintended consequences

A trial that recruited community‐based peer health workers consulting with clinic staff about adults who were receiving or started receiving antiretroviral therapy reported that healthcare professionals allocated to the intervention were not always able to charge the mobile phone, and that some mobile phones were stolen (Chang 2011; Analysis 15.1). Another trial where community nurses consulted with diabetes specialist nurses and podiatrists about adults with new diabetes‐related foot ulcers through videoconference, reported that images were not always transmitted (Taylor‐Gjevre 2018).

Equity considerations

Some of the included trials were designed and implemented to address geographical (Byamba 2015; Chang 2011; Davis 2003; Taylor‐Gjevre 2018) or socio‐economic limitations (Mansberger 2015; Sutherland 2009) on access to health care, and thus to promote equity for rural‐based and other disadvantaged populations who would have less access to health care (Table 5).

Even when a trial was specifically designed to address inequities identified a priori, it might still exclude the most vulnerable elements of the targeted population. Chang 2011 recruited peer health workers in rural Uganda, giving them access to experienced clinical staff through text messages and mobile phone calls, in order to provide better health care to HIV‐positive people. The peer health workers could interact with the participants using the mobile phone. The authors concluded that the relatively low penetration of mobile phones in Uganda, which at the time was 39%, alongside the challenges posed with phone‐charging in a setting where access to electricity is limited, might not only have limited the benefits of the intervention but also increased inequities. Furthermore, the authors also noted that the costs of the intervention could have been a limiting factor for the peer health workers, as the monthly stipend given for mobile phone credits was not always enough.

Whited 2013 excluded people who could not speak or read English, as well as those who failed a single‐question literacy assessment.
Gulacti 2017 assessed the use of a messaging system for communication between emergency physicians in the emergency department and physicians working elsewhere in the hospital, excluding consultants who did not own a smartphone with a secure messaging service.
Armstrong 2018 excluded people without access to the Internet and either a digital camera or a mobile phone with camera features.

Discussion

Summary of main results

We included 19 randomised trials of mobile technologies that recruited more than 5766 participants with varied conditions and health problems. Healthcare professionals included general practitioners, community‐based peer health workers, nurses and physiotherapists, who consulted with specialist healthcare professionals in another healthcare facility, and emergency physicians who consulted colleagues within the same facility. Most trials reported on the use of mobile technologies by general practitioners to consult with specialists, and reported that mobile technologies reduced the time between presentation and management of the health problem (4 trials, 656 participants; moderate‐certainty evidence). Accessing healthcare services through mobile technologies may reduce referrals and clinic visits among people with skin conditions and those with chronic kidney disease, and increase the likelihood of receiving an eye examination among people with diabetes and people referred for an ultrasound (9 trials, 4810 participants when reported, moderate‐certainty evidence). There was little evidence of a difference to patient‐reported quality of life outcomes (2 trials, 622 participants), clinician‐reported outcomes of disease progression (2 trials, 769 participants); or to healthcare providers and participants' satisfaction and acceptability, or cost (6 trials, 5423 participants, low‐certainty evidence). One trial reported on images being lost during transmission, when using mobile technologies and also in usual care; and one trial reported a few experiences of mobile phones not being charged or being lost. However, most trials did not measure or report technical problems.

Four studies reported on the use of mobile technologies by community health or home‐care workers to consult with clinic staff, there was little evidence of an effect on consultations in the trials that recruited participants with new diabetes‐related foot ulcer or older individuals treated with home enteral nutrition (2 trials, 370 participants; moderate‐certainty evidence). There was little or no difference for hospitalisations among older individuals treated with home enteral nutrition (1 trial, 188 participants; low‐certainty evidence), or mortality among people living with HIV or diabetes (2 trials, 1157 participants), for disease activity and health‐related quality of life in participants with rheumatoid arthritis (1 trial, 85 participants) or participant acceptability and satisfaction in people with new diabetes‐related foot ulcer or rheumatoid arthritis (178 participants).

Overall completeness and applicability of evidence

Most trials did not report data on providers' adherence, five trials reported on time between presentation and management of the health condition for the main comparison, and for the remaining comparisons and outcomes we identified very little evidence. A third of the trials recruited adults seeking care for dermatological conditions, reflecting current use of mobile technologies in healthcare settings.

The use of mobile technologies for communication between healthcare professionals and patient management might be particularly relevant for settings where there is a shortage of healthcare providers. However, most of the trials were conducted in high‐income (eleven trials in North America and six trials in Europe) or upper‐middle‐income countries (two trials, one in North America and one in Asia), with one trial each conducted in a lower‐middle rural country (Mongolia) and a low‐income country (Uganda). A similar range of countries was reported in a review of mobile technologies for healthcare service delivery processes (Free 2013). Specific challenges might arise when implementing trials in those contexts, such as the lack of access to power sockets to charge the mobile phones, highlighted by the peer health workers interviewed by Chang 2011 in Uganda but not in the study conducted in Mongolia.

Similar contextual factors might contribute to the applicability of the evidence. One factor often mentioned was the variation in healthcare professionals' willingness to use mHealth (Azogil‐López 2019; Liddy 2019a), to attend training (Liddy 2019a), to invite people to participate (Eminović 2009), to select participants for electronic referrals (Van Gelder 2017), or to hold face‐to‐face appointments (Iversen 2018), or provide the required feedback (Sutherland 2009). Four trials reported that recruited participants might not have been representative of the general population, as the study population was more educated (Armstrong 2018), more likely to be male (Mansberger 2015; Whited 2002; Whited 2013), and more likely to be healthier (Armstrong 2018; Mansberger 2015). There was also variation associated with participants' location, as those allocated to the intervention who lived closer to the referral setting were less likely to accept a telephone appointment (Azogil‐López 2019) and more likely to be referred to a face‐to‐face appointment from their healthcare provider (Iversen 2018). One trial reported that participants allocated to the control group had to travel on average 98 km to receive face‐to‐face care, thus indicating that the intervention might provide particular benefits in settings with a low‐density population (Byamba 2015).

Some of the included trials were designed and implemented to address geographical limitations on access to health care and thus allowed for healthcare providers who were geographically separated to exchange clinical information, promoting equity for rural‐based and other disadvantaged populations who would have less access to healthcare. Two trials were conducted in rural settings: South Carolina, USA (Davis 2003) and Saskatchewan, Canada (Taylor‐Gjevre 2018). Two trials recruited participants from socio‐economically disadvantaged areas (Mansberger 2015; Sutherland 2009).

Certainty of the evidence

The included randomised trials were mostly at low or unclear risk of selection bias. We downgraded the evidence for almost all of the outcomes due to a high risk of performance bias, and almost half of the trials were also at risk of detection, attrition, and reporting biases. We also downgraded some of the evidence for imprecision, due to the relatively small size of the trials. Our confidence in the effect estimates overall is moderate, although due to the relatively low number of trials, different uses of mHealth interventions and small numbers of participants recruited there is a possibility that the estimate of the effect is substantially different.

Potential biases in the review process

We limited the risk of publication bias by conducting a comprehensive literature search of different databases, including published articles, clinical trials registries and unpublished mHealth evidence. The WHO issued a call for papers through popular digital health communities of practice to identify additional primary trials as well as grey literature, all of which have contributed to limit publication bias. Two review authors screened records, extracted data and assessed the certainty of the evidence using GRADE, with discussion with the author team whenever there were any discrepancies.

Agreements and disagreements with other studies or reviews

Hasselberg 2014 conducted a review on image‐based medical expert teleconsultation, with 24 studies, including non‐randomised and feasibility studies. The overall results were similar to ours. A review on asynchronous electronic consultations that included 36 trials, seven of which were randomised trials, reported that healthcare providers were generally satisfied with the timely advice received and the health care provided to the participants (Liddy 2016). When updating the review, Liddy 2019b included non‐randomised evidence and concluded that eConsults were expanding beyond teledermatology and that providers from other specialties were also satisfied. We found limited evidence from randomised trials about how satisfied healthcare providers are with mHealth to communicate with other providers. For both reviews the authors concluded that there was limited research on morbidity and mortality, which is consistent with our results (Liddy 2016; Liddy 2019b).

A Cochrane qualitative evidence synthesis (QES) on healthcare providers' perceptions and experiences of using mHealth technologies to deliver primary care healthcare services found that while providers thought that mobile technologies improved their work and relationships with other providers as well as participants, they also highlighted specific challenges such as access to electricity and network coverage (Odendaal 2020). Similarly, an unpublished overview of factors influencing the acceptability, feasibility and implementation of mobile health technologies also reported problems with installation and usability, as well as issues with electricity and connection (Glenton 2019). This is consistent with our results, especially for settings where constant access to electricity might be an issue (Chang 2011).

Authors' conclusions

Implications for practiceMobile technologies are widespread, with the quality of transmission continuing to improve. Healthcare organisations in a number of settings have started to provide their healthcare providers with smartphones (Dala‐Ali 2011) and healthcare professionals often use their mobile phones to share clinical information, including the transmission of images (Mobasheri 2015). This review found that mobile technologies may reduce the time between presentation and management of the health condition when primary care providers or emergency physicians use them to consult with specialists, may increase the likelihood of receiving a clinical examination among participants with diabetes and those who required an ultrasound and may reduce referrals to secondary or tertiary care.One concern that has been raised is about data‐sharing and privacy (Chang 2011; Gulacti 2017; WHO 2011). Most of the included trials reported using secure web connections, and mobile phone applications are being developed for secure communications between medical staff at work. A recent review reported that the main barriers to the adoption of mHealth by healthcare professionals concern the perceived usefulness and ease of use, concerns surrounding privacy, security, and technological issues, cost, time, and how it will impact the interaction with colleagues, patients, and management (Gagnon 2016), even in areas where the use of mobile technologies is more common. Training is usually required to support implementation, for instance teledermatology has been implemented in several settings and its optimal implementation includes training of primary healthcare providers on how to use the mobile equipment to obtain high‐quality images (Kukutsch 2017); this was highlighted by some of the included trials (e.g. Eminović 2009; Piette 2017).There was little evidence about healthcare providers' satisfaction with the intervention in the trials we identified, and although healthcare providers reported that mobile technologies allowed for care to be delivered more quickly and facilitated triage, one study reported that they were less confident in their diagnosis and management plans when using teledermatology, compared with face‐to‐face care (Whited 2002). However, it is likely that this would improve with experience. A qualitative evidence synthesis reported that mobile technologies assisted contact with colleagues, and recommended that healthcare providers should be part of the planning, implementation, and evaluation of mobile health programmes. (Odendaal 2020). Similarly, it is important to establish whether mobile devices alleviate providers' workload, or instead add to it, including whether there is the capacity to provide the level of supervision and support required (Odendaal 2020).

Implications for researchFunding is required to support the conduct of randomised trials of mobile technology interventions in settings where these types of intervention may have the potential to significantly strengthen health systems, such as remote locations and where there is a shortage of specialist services.Process evaluations, conducted alongside randomised trials, to identify factors that might modify the effect of mHealth interventions in different contexts would be a valuable addition to the evidence base (Craig 2008). Identifying core outcomes might be a useful step, for example, understanding the impact of mHealth on providers' adherence to guidelines, time from presentation to resolution, and participants' health status and well‐being are outcomes for which more evidence is required. Research should also be conducted into consideration of factors to support implementation, such as the high attrition rates commonly found in studies that use mobile technologies.Detailed and standardised reporting of mobile health interventions, technical features and context will contribute to the quality of the evidence available (Agarwal 2016).

History

Protocol first published: Issue 1, 2018
Review first published: Issue 8, 2020

Notes

This review is based on standard text and guidance provided by Cochrane Effective Practice and Organisation of Care (EPOC).

Acknowledgements

AcknowledgementsWe acknowledge the help and support of Cochrane Effective Practice and Organisation of Care (EPOC), through the editorial input of the following editors and peer referees, who provided comments to improve the review: Julia Worswick (EPOC former Managing Editor); Paul Miller (EPOC information specialist); Joshua Vogel (EPOC contact editor); Michael Kent Ransom (EPOC internal editor); Craig Ramsay (EPOC stats editor); Andrew Farmer (external referee), and Elizabeth Shaw and Euphrasia Ebai‐Atuh (consumer referees). The authors would also like to thank John Eyers for designing and running the search strategies; Hanna Bergman, Anna Georgeson, Jennifer Petkovic, Nicola Maayan, and Rachel Richardson, from Cochrane Response, for screening and extracting data; and Kate Cahill for copy‐editing the review.National Institute for Health Research (NIHR), via Cochrane Infrastructure funding to the Effective Practice and Organisation of Care (EPOC) Group. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, National Health Service (NHS) or the Department of Health.We are grateful to the Guideline Development Group of the Digital Health Guidelines (World Health Organization) for their constructive feedback in formulating the guiding questions for this systematic review, and for the funding they provided to complete the review.Cochrane Response, an evidence services unit operated by the Cochrane Collaboration, provided screening and data extraction services for part of this review.

Competing interests

Declarations of interestARM: Consultancy from Infarmed ‐ national authority of medicines and health products. Health Technology Assessment Commission. Payment for development of education presentations from Portuguese Institute of Oncology ‐ Lisbon. BB: none known. CG: none known. DGB: "I was commissioned by the WHO to conduct this review." GM: owns stock in Apple Computer. GV: "Since October 2017 I have been employed by Cochrane Response, an evidence services unit operated by the Cochrane Collaboration and contracted by the WHO to produce this review." IRC: none known. MF: none known. NH: "Since June 2016 I have been employed by Cochrane Response, an evidence services unit operated by the Cochrane Collaboration and contracted by the WHO to produce this review". SL: "I am the Joint Co‐ordinating Editor for the Cochrane Effective Practice and Organisation of Care Review Group. I am also a member of the WHO Executive Guideline Steering Group on maternal and perinatal health recommendations". SS: "I am the Joint Co‐ordinating Editor for the Cochrane Effective Practice and Organisation of Care Review Group." TT: none known.

References

1. ArmstrongAW, ChambersCJ, MaverakisE, ChengMY, DunnickCA, Chren M-M, et al. Effectiveness of online vs in-person care for adults with psoriasis: a randomized controlled trial. JAMA Network Open2018;1(6):e183062. [DOI: 10.1001/jamanetworkopen.2018.3062]PMC632445330646223 doi:10.1001/jamanetworkopen.2018.3062 PubMed
2. YoungPM, ChenAY, FordAR, ChengMY, LaneCJ, ArmstrongAW. Effects of online care on functional and psychological outcomes in patients with psoriasis: a randomized controlled trial. Journal of the American Academy of Dermatology 2019 May 30 [Epub ahead of print]. [DOI: 10.1016/j.jaad.2019.05.089]31175908 doi:10.1016/j.jaad.2019.05.089 PubMed
3. Azogil-LópezLM, Pérez-LázaroJJ, Ávila-PecciP, Medrano-SanchézEM, Coronado-VázquezMV. Effectiveness of a new model of telephone derivation shared between primary care and hospital care [Efectividad de un nuevo modelo de derivación telefónica compartida entre atención primaria e atención hospitalaria]. Atención Primaria2019;51(5):278-84. [DOI: 10.1016/j.aprim.2018.02.006]PMC683699729699717 doi:10.1016/j.aprim.2018.02.006 PubMed
4. Azogil-LópezLM, Pérez-LázaroJJ, Medrano-SanchézEM, Goméz-SalgadoJ, Coronado-VázquezV. DETELPROG study: Effectiveness of a new model of scheduled telephone referral from primary care to internal medicine. A randomised controlled study. Journal of Clinical Medicine2019;8(5):688. [DOI: 10.3390/jcm8050688]PMC657218631100835 doi:10.3390/jcm8050688 PubMed
5. ByambaK, Syed-AbdulS, García-RomeroM, Huang C-W, NergyiS, NyamdorjA, et al. Mobile teledermatology for a prompter and more efficient dermatological care in rural Mongolia. British Journal of Dermatology2015;173(1):265-7. [DOI: 10.1111/bjd.13607]PMC567921625494968 doi:10.1111/bjd.13607 PubMed
6. ChangLW, KagaayiJ, AremH, NakigoziG, SsempijjaV, SerwaddaD, et al. Impact of a mHealth intervention for peer health workers on AIDS care in rural Uganda: a mixed methods evaluation of a cluster-randomized trial. AIDS Behavior2011;15(8):1776-84. [DOI: 10.1007/s10461-011-9995-x]PMC326575221739286 doi:10.1007/s10461-011-9995-x PubMed
7. ChangLW, KagaayiJ, NakigoziG, PackerAH, SerwaddaD, QuinnTC, et al. Responding to the human resource crisis: Peer health workers, mobile phones, and HIV care in Rakai, Uganda. AIDS Patient Care and STDs2008;22(3):173-4. [DOI: 10.1089/apc.2007.0234]10.1089/apc.2007.0234PMC267457218290750 doi:10.1089/apc.2007.0234 PubMed
8. Chang LW1, KagaayiJ, NakigoziG, SerwadaD, QuinnTC, GrayRH, et al. Cost analyses of peer health worker and mHealth support interventions for improving AIDS care in Rakai, Uganda. AIDS Care2013;25(5):652-6. [DOI: 10.1080/09540121.2012.722600]PMC377347222971113 doi:10.1080/09540121.2012.722600 PubMed
9. DavisRM, FowlerS, BellisK, PocklJ, Al PakalnisV, WoldorfA. Telemedicine improves eye examination rates in individuals with diabetes: a model for eye-care delivery in underserved communities. Diabetes Care2003;26(8):2476. 10.2337/diacare.26.8.247612882889 doi:10.2337/diacare.26.8.2476 PubMed
10. DavisRM, PocklJ, BellisK. Improved diabetic eye care utilizing telemedicine: a randomized controlled trial. Investigative Ophthalmology & Visual Science2003;44:166.
11. EminovićN, De KeizerNF, WyattJC, Ter RietG, PeekN, Van WeertHC, et al. Teledermatologic consultation and reduction in referrals to dermatologists: a cluster randomized controlled trial. Archives of Dermatology2009;145(5):558-64. [DOI: 10.1001/archdermatol.2009.44]19451500 doi:10.1001/archdermatol.2009.44 PubMed
12. EminovićN, DijkgraafMG, BerghoutRM, PrinsAH, Bindels, PJ, De KeizerNF. A cost minimisation analysis in teledermatology: Model-based approach. BMC Health Services Research2011;10:251. [DOI: 10.1186/1472-6963-10-251]PMC294087920738871 doi:10.1186/1472-6963-10-251 PubMed
13. GulactiU, LokU. Comparison of secure messaging application (WhatsApp) and standard telephone usage for consultations on length of stay in the ED. A prospective randomized controlled study. Applied Clinical Informatics2017;8(3):742-53. [DOI: 10.4338/ACI-2017-04-RA-0064]PMC622069328880047 doi:10.4338/ACI-2017-04-RA-0064 PubMed
14. IversenMM, EspehaugB, HauskenMF, GraueM, ØstbyeT, SkeieS, et al. Telemedicine versus standard follow-up care for diabetes-related foot ulcers: Protocol for a cluster randomized controlled noninferiority trial (DiaFOTo). JMIR Research protocols2016;5(3):e148. [DOI: 10.2196/resprot.5646]PMC496955027430301 doi:10.2196/resprot.5646 PubMed
15. KolltveitBH, ThorneS, GraueM, GjengedalE, IversenMM, KirkevoldM. Telemedicine follow-up facilitates more comprehensive diabetes foot ulcer care: a qualitative study in home-based and specialist health care. Journal of Clinical Nursing2017;27(5-6):e1134-45. [DOI: 10.1111/jocn.14193]10.1111/jocn.1419329193527 doi:10.1111/jocn.14193 PubMed
16. Smith-StrømH, IglandJ, ØstbyeT, TellGS, Hausken MF4, GraueM, et al. The effect of telemedicine follow-up care on diabetes-related foot ulcers: a cluster-randomized controlled noninferiority trial. Diabetes Care2018;5(3):e148. [DOI: 10.2196/resprot.5646]29187423 doi:10.2196/resprot.5646 PubMed
17. Smith-StrømH, IversenMM, GraueM, SkeieS, KirkevoldM. An integrated wound-care pathway, supported by telemedicine, and competent wound management: Essential in follow-up care of adults with diabetic foot ulcers. International Journal of Medical informatics2016;64:59-66. [DOI: 10.1016/j.ijmedinf.2016.06.020]27573312 doi:10.1016/j.ijmedinf.2016.06.020 PubMed
18. LiddyC, MarangerJ, AfkhamA, KeelyE. Ten steps to establishing an e-consultation service to improve access to specialist care. Telemedicine Journal and E-Health2013;19(12):982-90. [DOI: 10.1089/tmj.2013.0056]PMC385043424073898 doi:10.1089/tmj.2013.0056 PubMed
19. LiddyC, MorozI, KeelyE, TaljaardM, ArmstrongCD, AfkhamA, et al. Understanding the impact of a multispecialty electronic consultation service on family physician referral rates to specialists: a randomized controlled trial using health administrative data. Trials2019;20:348. [DOI: 10.1186/s13063-019-3393-5]PMC655885031182123 doi:10.1186/s13063-019-3393-5 PubMed
20. MansbergerSL, GleitsmannK, GardinerS, ShepplerC, DemirelS, WootenK, et al. Comparing the effectiveness of telemedicine and traditional surveillance in providing diabetic retinopathy screening examinations: a randomized controlled trial. Telemedicine Journal and e-Health2013;19(12):942-48. [DOI: 10.1089/tmj.2012.0313]PMC385042824102102 doi:10.1089/tmj.2012.0313 PubMed
21. MansbergerSL, ShepplerC, BarkerG, GardinerSK, DemirelS, WootenK, et al. Long-term comparative effectiveness of telemedicine in providing diabetic retinopathy screening examinations: a randomized clinical trial. JAMA Ophtalmology2015;133(5):518-25. [DOI: 10.1001/jamaophthalmol.2015.1]PMC602385525741666 doi:10.1001/jamaophthalmol.2015.1 PubMed
22. OrlandoniP, Jukic PeladicN, SpazzafumoL, VenturiniC, ColaC, SparvoliD, et al. Utility of video consultation to improve the outcomes of home enteral nutrition in a population of frail older patients. Geriatrics & Gerontology International2016;16(6):762-7. [DOI: 10.1111/ggi.12551]26171754 doi:10.1111/ggi.12551 PubMed
23. PakH, TriplettCA, LindquistJH, GrambowSC, WhitedJD. Store-and-forward teledermatology results in similar clinical outcomes to conventional clinic-based care. Journal of Telemedicine and Telecare2007;13(1):26-30. [DOI: 10.1258/135763307779701185]17288655 doi:10.1258/135763307779701185 PubMed
24. PakHS, DattaSK, TriplettCA, LindquistJH, GrambowSC, WhitedJD. Cost minimization analysis of a store-and-forward teledermatology consult system. Telemedicine Journal and e-Health2009;15(2):160-5. [DOI: 10.1089/tmj.2008.0083]19292625 doi:10.1089/tmj.2008.0083 PubMed
25. PietteE, NougairèdeM, VuongV, CrickxB, TranVT. Impact of a store-and-forward teledermatology intervention versus usual care on delay before beginning treatment: a pragmatic cluster-randomized trial in ambulatory care. Journal of Telemedicine and Telecare2017;23(8):725-32. [DOI: 10.1177/1357633X16663328]27496852 doi:10.1177/1357633X16663328 PubMed
26. RordanJ, OttenritterC, SullivanK, DaSilvaK, O'ConnorD, DayalA. MedLibs: a mobile application for facilitating emergency department consultation requests. Annals of Emergency Medicine2015;66(4S):S75.
27. SutherlandJE, SutphinHD, RawlinsF, RedicanK, BurtonJ. A comparison of telesonography with standard ultrasound care in a rural Dominican clinic. Journal of Telemedicine and Telecare2009;15(4):191-5. [DOI: 10.1258/jtt.2009.080909]19471031 doi:10.1258/jtt.2009.080909 PubMed
28. Taylor-GjevreR, NairB, BathB, OkpalauwaekweU, SharmaM, PenzE, et al. Addressing rural and remote access disparities for patients with inflammatory arthritis through video-conferencing and innovative inter-professional care models. Musculoskeletal Care2018;16(1):90-5. [DOI: 10.1002/msc.1215]28967235 doi:10.1002/msc.1215 PubMed
29. Van GelderVA, Scherpbier-de HaanND, Van BerkelS, AkkermansRP, De GrauwIS, AdangEM, et al. Web-based consultation between general practitioners and nephrologists: a cluster randomized controlled trial. Family Practice2017;34(4):430-6. [DOI: 10.1093/fampra/cmw131]28158576 doi:10.1093/fampra/cmw131 PubMed
30. WhitedJD, DattaS, HallRP, FoyME, MarbreyLE, GrambowSC, et al. An economic analysis of a store and forward teledermatology consult system. Telemedicine Journal and e-Health2003;9(4):351-60. [DOI: 10.1089/153056203772744671]14980092 doi:10.1089/153056203772744671 PubMed
31. WhitedJD, HallRP, FoyME, MarbreyLE, GrambowSC, DudleyTK, et al. Patient and clinician satisfaction with a store-and-forward teledermatology consult system. Telemedicine Journal and e-Health2004;10(4):422-31. [DOI: 10.1089/tmj.2004.10.422]15689645 doi:10.1089/tmj.2004.10.422 PubMed
32. WhitedJD, HallRP, FoyME, MarbreyLE, GrambowSC, DudleyTK, et al. Teledermatology's impact on time to intervention among referrals to a dermatology consult service. Telemedicine Journal and e-Health2002;8(3):313-21. [DOI: 10.1089/15305620260353207]10.1089/1530562026035320712419025 doi:10.1089/15305620260353207 PubMed
33. DattaSK, WarshawEM, EdisonKE, KapurK, ThottapurathuL, MoritzTE, et al. Cost and utility analysis of a store-and-forward teledermatology referral system: a randomized clinical trial. JAMA Dermatology2015;151(12):1323-9. [DOI: 10.1001/jamadermatol.2015.2362]26375589 doi:10.1001/jamadermatol.2015.2362 PubMed
34. WhitedJD, WarshawEM, EdisonKE, KapurK, ThottapurathuL, RajuS, et al. Effect of store and forward teledermatology on quality of life: a randomized controlled trial. JAMA Dermatology2013;149(5):584-91. [DOI: 10.1001/2013.jamadermatol.380]PMC372619923426111 doi:10.1001/2013.jamadermatol.380 PubMed
35. WhitedJD, WarshawEM, KapurK, EdisonKE, ThottapurathuL, RajuS, et al. Clinical course outcomes for store and forward teledermatology versus conventional consultation: a randomized trial. Journal of Telemedicine and Telecare2013;19(4):197-204. [DOI: 10.1177/1357633X13487116]23666440 doi:10.1177/1357633X13487116 PubMed
36. AteudjieuJ, StollB, Nguefack-TsagueG, TchangouC, GentonB. Vaccines safety: effect of supervision or SMS on reporting rates of adverse events following immunization (AEFI) with meningitis vaccine (MenAfriVac™): a randomized controlled trial. Vaccine2014;32(43):5662-8. 10.1016/j.vaccine.2014.08.01225160064 doi:10.1016/j.vaccine.2014.08.012 PubMed
37. AtnafuA, OttoK, HerbstCH. The role of mHealth intervention on maternal and child health service delivery: findings from a randomized controlled field trial in rural Ethiopia. mHealth2017;3:39. 10.21037/mhealth.2017.08.04PMC568238729184891 doi:10.21037/mhealth.2017.08.04 PubMed
38. Batista JdA, FurtadoMV, KatzN, AgostinhoMR, NetoBS, HarzheimE, et al. Telemedicine-supported transition of stable coronary artery disease patients from tertiary to primary health care facilities: protocol for a randomized non-inferiority trial. BMC Health Services Research2016;16:227. 10.1186/s12913-016-1469-4PMC493758127387752 doi:10.1186/s12913-016-1469-4 PubMed
39. BettinelliM, LeiY, BeaneM, MackeyC, LieschingTN. Does robotic telerounding enhance nurse-physician collaboration satisfaction about care decisions?Telemedicine Journal and e-Health2015;21(8):637-43. [DOI: 10.1089/tmj.2014.0162]25938296 doi:10.1089/tmj.2014.0162 PubMed
40. BurnsCL, KularatnaS, WardEC, HillAJ, ByrnesJ, KennyLM. Cost analysis of a speech pathology synchronous telepractice service for patients with head and neck cancer. Head & Neck2016;39(12):2470-80. 10.1002/hed.2491628963804 doi:10.1002/hed.24916 PubMed
41. BuvikA, BuggeE, KnutsenG, SmåbrekkeA, WilsgaardT. Quality of care for remote orthopaedic consultations using telemedicine: a randomised controlled trial. BMC Health Services Research2016;16:483. 10.1186/s12913-016-1717-7PMC501704527608768 doi:10.1186/s12913-016-1717-7 PubMed
42. ChiaravallotiA, SchunckCH, SørensenTF, ThestrupJ, RosengrenP, PellicanoC, et al. PICASO: A personalised integrated care platform. Clinical and Translational Imaging2017;5(1):S141.
43. ConlinPR, FischBM, CavalleranoAA, CavalleranoJD, BursellSE, AielloLM. Nonmydriatic teleretinal imaging improves adherence to annual eye examinations in patients with diabetes. Journal of Rehabilitation Research and Development2006;43(6):733-40. 10.1682/jrrd.2005.07.011717310422 doi:10.1682/jrrd.2005.07.0117 PubMed
44. Da SilvaR, RadosD, Dos SantosE, KatzN, HarzheimE, PolanczykC, et al. Teleconsultation support patients with benign prostatic hyperplasia being discharged from specialized care: a randomized noninferiority study. Journal of Urology2018;199(4):e686-7.
45. FerrándizL, Ojeda-VilaT, CorralesA, Martín-GutiérrezFJ, Ruíz-de-CasasA, GaldeanoR, et al. Impact of dermoscopy on an internet-based skin cancer triage system: interim results of a randomized study. Journal of the American Academy of Dermatology2017;76(2):342-3. 10.1016/j.jaad.2016.02.116528088994 doi:10.1016/j.jaad.2016.02.1165 PubMed
46. FerrándizL, Ojeda-VilaT, CorralesA, Martín-GutiérrezFJ, Ruíz-de-CasasA, GaldeanoR, et al. Internet-based skin cancer screening using clinical images alone or in conjunction with dermoscopic images: a randomized teledermoscopy trial. Journal of the American Academy of Dermatology2017;76(4):676-82. 10.1016/j.jaad.2016.10.04128089728 doi:10.1016/j.jaad.2016.10.041 PubMed
47. GolbersteinE, KolvenbachS, CarruthersH, DrussB, GoeringP. Effects of electronic psychiatric consultations on primary care provider perceptions of mental health care: Survey results from a randomized evaluation. Healthcare2017;S2213-0764(16):30173-7. [DOI: 10.1016/j.hjdsi.2017.01.002]28162990 doi:10.1016/j.hjdsi.2017.01.002 PubMed
48. GongE, GuW, SunC, TurnerEL, ZhouY, Li z, et al. System-integrated technology-enabled model of care to improve the health of stroke patients in rural China: protocol for SINEMA—a cluster-randomized controlled trial. American Heart Journal2019;207:27-39. 10.1016/j.ahj.2018.08.01530408621 doi:10.1016/j.ahj.2018.08.015 PubMed
49. HaridyJ, IyngkaranG, TseE. An eHealth model of care for community hepatitis c management: The healthelink project. Journal of Gastroenterology and Hepatology2017;32:70-71.
50. LoaneMA, BloomerSE, CorbettR, EedyDJ, EvansC, HicksN, et al. A randomized controlled trial assessing telehealth economics of realtime teledermatology compared with conventional care: an urban versus rural perspective. Journal of Telemedicine and Telecare2001;7(2):108-18. [DOI: 10.1258/1357633011936246]11331049 doi:10.1258/1357633011936246 PubMed
51. LoaneMA, BloomerSE, CorbettR, EedyDJ, HicksN, LoteryHE, et al. A randomized controlled trial to assess the clinical effectiveness of both realtime and store-and-forward teledermatology compared with conventional care. Journal of Telemedicine and Telecare2000;6(Suppl. 1):S1-3. 10.1258/135763300193395210793956 doi:10.1258/1357633001933952 PubMed
52. WoottonR, BloomerSE, CorbettR, EedyDJ, HicksN, LoteryHE, et al. Multicentre randomised control trial comparing real time teledermatology with conventional outpatient dermatological care: societal cost-benefit analysis. BMJ2000;320(7244):1252-6. 10.1136/bmj.320.7244.1252PMC2737010797038 doi:10.1136/bmj.320.7244.1252 PubMed
53. NCT02710799. Evaluation of the effects of teleconsultations on a endocrinology referral list. clinicaltrials.gov/ct2/show/NCT02710799 (first received 17 March 2016).
54. AndersonD, VillagraV, ComanEN, ZlatevaI, HutchinsonA, VillagraJ, et al. A cost-effectiveness analysis of cardiology eConsults for Medicaid patients. American Journal of Managed Care2018;24(1):e9-e16. 29350511 PubMed
55. Nwando OlayiwolaJ, AndersonD, JepealN, AseltineR, PickettC, YanJ, et al. Electronic consultations to improve the primary care-specialty care interface for cardiology in the medically underserved: a cluster-randomized controlled trial. Annals of Family Medicine2016;14(2):133-40. [DOI: 10.1370/afm.1869]PMC478151626951588 doi:10.1370/afm.1869 PubMed
56. LoaneMA, OakleyA, RademakerM, BradfordN, FleischlP, KerrP, et al. A cost-minimization analysis of the societal costs of realtime teledermatology compared with conventional care: results from a randomized controlled trial in New Zealand. Journal of Telemedicine and Telecare2001;7(4):233-8. [DOI: 10.1258/1357633011936453]11506759 doi:10.1258/1357633011936453 PubMed
57. OakleyAM, KerrP, DuffillM, RademakerM, FleischlP, BradfordN, et al. Patient cost-benefits of realtime teledermatology: a comparison of data from Northern Ireland and New Zealand. Journal of Telemedicine and Telecare2000;6(2):97-101. [DOI: 10.1258/1357633001935112]10824377 doi:10.1258/1357633001935112 PubMed
58. OwenRR, WoodwardEN, DrummondKL, DeenTL, OliverKA, PetersenNJ, et al. Using implementation facilitation to implement primary care mental health integration via clinical video telehealth in rural clinics: protocol for a hybrid type 2 cluster randomized stepped-wedge design. Implementation Science2019;14(1):33. 10.1186/s13012-019-0875-5PMC642982330898129 doi:10.1186/s13012-019-0875-5 PubMed
59. PhillipsJL, HenekaN, LovellM, LamL, DavidsonP, BoyleF, et al. A phase III wait-listed randomised controlled trial of novel targeted inter-professional clinical education intervention to improve cancer patients' reported pain outcomes (The Cancer Pain Assessment (CPAS) Trial): study protocol. Trials2019;20(1):62. 10.1186/s13063-018-3152-zPMC633928330658657 doi:10.1186/s13063-018-3152-z PubMed
60. PrzybyloJA, WangA, LoftusP, EvansKH, ChuI, ShiehL. Smarter hospital communication: Secure smartphone text messaging improves provider satisfaction and perception of efficacy, workflow. Journal of Hospital Medicine2014;9(9):573-8. [DOI: 10.1002/jhm.2228]PMC426315725110991 doi:10.1002/jhm.2228 PubMed
61. RomeroG, SánchezP, GarcíaM, CortinaP, VeraE, GarridoJA. Randomized controlled trial comparing store-and-forward teledermatology alone and in combination with web-camera videoconferencing. Clinical and Experimental Dermatology2010;35(3):311-7. [DOI: 10.1111/j.1365-2230.2009.03503]19874323 doi:10.1111/j.1365-2230.2009.03503 PubMed
62. WesargT, WasowskiA, SkarzynskiH, RamosA, Falcon GonzalezJC, KyriafinisG, et al. Remote fitting in Nucleus cochlear implant recipients. Acta Oto-Laryngologica2010;130(12):1379-88. [DOI: 10.3109/00016489.2010.492480]20586675 doi:10.3109/00016489.2010.492480 PubMed
63. ACTRN12617000389303. Establishing the role of teleconsulting in the care of chronic conditions in rural areas of the Southern District Health Board (SDHB): a randomised controlled trial (RCT) in patients with inflammatory bowel disease. www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=372543 (first received 10 March 2017).
64. ACTRN12618001007224. A prospective randomised controlled study of telehealth specialist palliative care consultations in rural and metropolitan settings and the impact on patient and carer clinical outcomes and quality-of-life. www.anzctr.org.au/Trial/Registration/TrialReview.aspx?ACTRN=12618001007224 (first received 25 May 2018).
65. DoneN, OhDH, WeinstockMA, WithedJD, JacksonGL, KingHA, et al. VA Telederm study: protocol for a stepped-wedge cluster randomised trial to compare access to care for a mobile app versus a workstation-based store-and-forward teledermatology process. BMJ Open2018;8:e022218. 10.1136/bmjopen-2018-022218PMC630358830552249 doi:10.1136/bmjopen-2018-022218 PubMed
66. NCT03241589. Teledermatology mobile apps. clinicaltrials.gov/ct2/show/NCT03241589 (first received 7 August 2017).
67. FerruaM, Di PalmaM, LemareF, FourcadeA, LallouéB, Daumas-YatimF, et al. Impact of a cancer care coordination program based on health information technologies for patients treated by oral anticancer therapy: The CAPRI randomized trial. Annals of Oncology2017;28(Suppl 5):mdx388.066.
68. Gervès-PinquiéC, Daumas-YatimF, LallouéB, GiraultA, FerruaM, FourcadeA, et al. Impacts of a navigation program based on health information technology for patients receiving oral anticancer therapy: The CAPRI randomized controlled trial. BMC Health Services Research2017;17(1):133. 10.1186/s12913-017-2066-xPMC530787928193214 doi:10.1186/s12913-017-2066-x PubMed
69. NCT02828462. Impact of a monitoring device for patients with cancer treated using oral therapeutics (CAPRI). clinicaltrials.gov/ct2/show/NCT02828462 (first received 11 July 2016).
70. JeandidierN, ChaillousL, FrancS, BenhamouPY, SchaepelynckP, HanaireH, et al. DIABEO app software and telemedicine versus usual follow-up in the treatment of diabetic patients: protocol for the TELESAGE randomized controlled trial. JMIR Research Protocols2018;7(4):e66. 10.2196/resprot.9154PMC593453329674306 doi:10.2196/resprot.9154 PubMed
71. NCT02287532. Evaluation of the DIABEO system in poorly controlled DM1 or DM2 patients treated with a basal-bolus Insulin regimen (TELESAGE). clinicaltrials.gov/ct2/show/NCT02287532 (first received 10 November 2014).
72. KällanderK, StrachanD, SoremekunS, HillZ, LingamR, TibenderanaJ, et al. Evaluating the effect of innovative motivation and supervision approaches on community health worker performance and retention in Uganda and Mozambique: study protocol for a randomised controlled trial. Trials2015;16:157. [DOI: 10.1186/s13063-015-0657-6]PMC443298125873093 doi:10.1186/s13063-015-0657-6 PubMed
73. DRKS00012944. Implementation of teledermatologic referrals into general practice: A cluster-randomized controlled trial. www.drks.de/drks_web/setLocale_EN.do (first received 31 August 2017).
74. KochR, PolancA, HaumannH, KirtschigG, MartusP, ThiesC, et al. Improving cooperation between general practitioners and dermatologists via telemedicine: study protocol of the cluster-randomized controlled TeleDerm study. Trials 19;1(583):10.1186/s13063-018-2955-2. 10.1186/s13063-018-2955-2PMC620150830355358 doi:10.1186/s13063-018-2955-2 PubMed
75. NakayamaM, InoueR. Prospective randomized trial of telemedicine-based collaborativecare using a prefectural medical information network system. European Journal of Epidemiology2016;31(Suppl. 1):S141.
76. NCT02821143. The impact of telemedicine to support palliative care resident in nursing home (TELESM). clinicaltrials.gov/ct2/show/NCT02821143 (first received 1 July 2016).
77. NCT02986256. Evaluation of the management of diabetic foot ulcers by telemedicine on the number of hospital days in diabetic patients (TELEPIED). clinicaltrials.gov/ct2/show/NCT02986256 (first received 8 December 2016).
78. NCT03137511. Optimizing Access to Care Through New Technologies: A randomized study evaluating the impact of telephone contact and the sending by the general practitioner of suspicious lesions melanoma photographs taken with a smartphone, on the time limit to the consultation with a dermatologist (OASE Melanome). clinicaltrials.gov/ct2/show/NCT03137511 (first received 2 May 2017).
79. NCT03559712. Effectiveness of collaborative tele-mental health services for ADHD in primary care: a randomized trial in Dubai (ECTSAP- Dubai Trial). clinicaltrials.gov/ct2/show/NCT03559712 (first received 18 June 2018).
80. NCT03662256. Reducing childhood hearing loss in rural Alaska through a preschool screening and referral process using mobile health and telemedicine. clinicaltrials.gov/ct2/show/NCT03662256 (first received 7 September 2018).
81. NCT02617875. Telemedical support for prehospital emergency medical service (TEMS). clinicaltrials.gov/ct2/show/NCT02617875 (first received 1 December 2015).
82. StevanovicA, BeckersSK, CzaplikM, BergrathS, CoburnM, BrokmannCB, et al. Telemedical support for prehospital Emergency Medical Service (TEMS trial): study protocol for a randomized controlled trial. Trials2017;18(1):43. [DOI: 10.1186/s13063-017-1781-2]PMC527033928126019 doi:10.1186/s13063-017-1781-2 PubMed
83. XuL, FangWY, ZhuF, ZhangHG, LiuK. A coordinated PCP-Cardiologist Telemedicine Model (PCTM) in China's community hypertension care: study protocol for a randomized controlled trial. Trials2017;18(1):236. [DOI: 10.1186/s13063-017-1970-z]PMC544530628545514 doi:10.1186/s13063-017-1970-z PubMed
84. American Academy of Pediatrics Comittee on Pediatric Workforce. The use of telemedicine to address access and physician workforce shortages. Pediatrics2015;136(1):202-9.10.1542/peds.2015-125326122802 doi:10.1542/peds.2015-1253 PubMed
85. AcetoG, PersicoV, PescapéA. The role of Information and Communication Technologies in healthcare: taxonomies, perspectives, and challenges. Journal of Network and Computer Applications2018;107:125-54. [DOI: 10.1016/j.jnca.2018.02.008] doi:10.1016/j.jnca.2018.02.008
86. Asia eHealth Information Network. Getting to know the Network. www.aehin.org/AboutUs.aspx (accessed 24 April 2020).
87. AgarwalS, LeFevreAE, LeeJ, L'EngleK, MehlG, SinhaC, et al, WHO mHealth Technical Evidence Review Group. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ2016;352:i1174. [DOI: 10.1136/bmj.i1174]26988021 doi:10.1136/bmj.i1174 PubMed
88. AgboolaSO, BatesDW, KvedarJC. Digital health and patient safety. JAMA2016;315(16):1697-8.10.1001/jama.2016.240227115372 doi:10.1001/jama.2016.2402 PubMed
89. AKDN Digital Health Programme. akdnehrc.org/ehealth_programme/ (accessed 31 July 2019).
90. ArainM, CampbellMJ, CooperCL, LancasterGA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Medical Research Methodology2010;10:67.10.1186/1471-2288-10-67PMC291292020637084 doi:10.1186/1471-2288-10-67 PubMed
91. BorensteinM, HedgesLV, HigginsJP, RothsteinHR. When does it make sense to perform a meta-analysis? In: Introduction to Meta-Analysis. Chichester (UK): John Wiley & Sons, Ltd, 2009.
92. Campbell M , Grimshaw J , Steen N, for the Changing Professional Practice Europe Group (EU BIOMED II Concerted Action). Sample size calculations for cluster randomized trials. Journal of Health Services Research & Policy2000;5(1):12-6.10.1177/13558196000050010510787581 doi:10.1177/135581960000500105 PubMed
93. CampbellJ, DussaultG, BuchanJ, Pozo-MartinF, Guerra AriasM, LeoneC, et al. A Universal Truth: No Health Without a Workforce. Recife, Brazil: Third Global Forum on Human Resources for Health; 2013. Forum report. Geneva: Global Health Workforce & World Health Organization, 2013.
94. CoeiraE, AshJ, BergM. The unintended consequences of Health Information Technology revisited. Yearbook of Medical Informatics2016;2016:163-9. [DOI: 10.15265/IY-2016-014]PMC517157627830246 doi:10.15265/IY-2016-014 PubMed
95. Covidence. Melbourne, Australia: Veritas Health Innovation, Version accessed 11 February 2020. Available at covidence.org.
96. CraigP, DieppeP, MacintyreS, MichieS, NazarethI, PetticrewM, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ2008;337:a1655. [DOI: <10.1136/bmj.a1655]10.1136/bmj.a1655PMC276903218824488 doi:10.1136/bmj.a1655 PubMed
97. Dala-AliBM, LloydMA, Al-AbedY. The uses of the iPhone for surgeons. Surgeon2011;9(1):44-8. [DOI: 10.1016/j.surge.2010.07.014]21195331 doi:10.1016/j.surge.2010.07.014 PubMed
98. DeeksJJ, HigginsJP, AltmanDG, editor(s), on behalf of the Cochrane Statistical Methods Group. Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JP, Churchill R, Chandler J, Cumpston MS, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017), The Cochrane Collaboration, 2017. Available from www.training.cochrane.org/handbook.
99. DistillerSR. Ottawa, Canada: Evidence Partners, Version accessed prior to 1 January 2019. Available at www.evidencepartners.com/products/distillersr-systematic-review-software/.
100. Cochrane Effective Practice and Organisation of Care (EPOC). Data collection form. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed 31 August 2017).
101. Cochrane Effective Practice and Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed 31 August 2017).
102. Cochrane Effective Practice and Organisation of Care (EPOC). EPOC worksheets for preparing a 'Summary of findings' table using GRADE. EPOC resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed 31 August 2017).
103. Cochrane Effective Practice and Organisation of Care (EPOC). Reporting the effects of an intervention in EPOC reviews. EPOC Resources for review authors, 2017. epoc.cochrane.org/resources/epoc-resources-review-authors (accessed 31 August 2017).
104. FreeC, PhillipsG, WatsonL, GalliL, FelixL, EdwardsP, et al. The effectiveness of mobile-health technologies to improve health care service delivery processes: a systematic review and meta-analysis. PLOS Medicine2013;10(1):e1001363. [DOI: 10.1371/journal.pmed.1001363]PMC356692623458994 doi:10.1371/journal.pmed.1001363 PubMed
105. GagnonMP, NgangueP, Payne-GagnonJ, DesmartisM. m-Health adoption by healthcare professionals: a systematic review. JAMA2016;23(1):212-20. [DOI: 10.1093/jamia/ocv052]PMC781491826078410 doi:10.1093/jamia/ocv052 PubMed
106. GlentonC, AkamaE, TamratT. Factors influencing the acceptability, feasibility and implementation of interactive telemedicine: an overview of reviews. In: WHO Guideline: Recommendations on Digital Interventions for Health System Strengthening Web Supplement 2: Summary of Findings and GRADE Tables. Geneva: World Health Organization, 2019. [Licence: CC BY-NC-SA 3.0 IGO]
107. Gonçalves-BradleyDC, BuckleyBS, FønhusMS, GlentonC, HenschkeN, LewinS, et al. Mobile-based technologies to support client to healthcare provider communication and management of care. Cochrane Database of Systematic Reviews2018, Issue 1. Art. No: CD012928. [DOI: 10.1002/14651858.CD012928]PMC743739232813281 doi:10.1002/14651858.CD012928 PubMed
108. GRADEpro GDT. Version accessed prior to 20 December 2017. Hamilton (ON): McMaster University (developed by Evidence Prime), 2015.
109. GuyattGH, OxmanAD, VistG, KunzR, Falck-YtterY, Alonso-CoelloP, et al, GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ2008;336(7650):924-6.10.1136/bmj.39489.470347.ADPMC233526118436948 doi:10.1136/bmj.39489.470347.AD PubMed
110. HasselbergM, BeerN, BlomL, WallisLA, LaflammeL. Image-based medical expert teleconsultation in acute care of injuries: a systematic review of effects on information accuracy, diagnostic validity, clinical outcome, and user satisfaction. PLOS One2014;9(6):e98539. [DOI: 10.1371/journal.pone.0098539]PMC404189024887257 doi:10.1371/journal.pone.0098539 PubMed
111. HigginsJP, AltmanDG, SterneJA, editor(s). Chapter 8: Assessing risk of bias in included studies. in: Higgins JP, Churchill R, Chandler J, Cumpston MS, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). The Cochrane Collaboration, 2017. Available from www.training.cochrane.org/handbook.
112. HigginsJP, ThomasJ, ChandlerJ, CumpstonM, LiT, PageMJ, et al editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 6.0 (updated July 2019). The Cochrane Collaboration, 2019. Available from www.training.cochrane.org/handbook.
113. International Telecomunications Union. Measuring digital development: Facts and figures 2019. https://www.itu.int/en/ITU-D/Statistics/Documents/statistics/2019/ITU_Key_2005-2019_ICT_data_with%20LDCs_28Oct2019_Final.xls (accessed 27 April 2020).
114. KonradTR, WilliamsES, LinzerM, McMurrayJ, PathmanDE, GerrityM, et al. Measuring physician job satisfaction in a changing workplace and a challenging environment. Medical Care1999;37(11):1174-82.10.1097/00005650-199911000-0001010549620 doi:10.1097/00005650-199911000-00010 PubMed
115. KruseCS, KrowskiN, RodriguezB, TranL, VelaJ, BrooksM. Telehealth and patient satisfaction: a systematic review and narrative analysis. BMJ Open2017;7(8):e016242.10.1136/bmjopen-2017-016242PMC562974128775188 doi:10.1136/bmjopen-2017-016242 PubMed
116. KukutschNA, ArgenzianoG, BergmanW. The opinion of dermoscopy experts about teledermoscopy involving primary care physicians and dermatologists. JEADV2017;31(10):e470-1.10.1111/jdv.1425928380269 doi:10.1111/jdv.14259 PubMed
117. KällanderK, TibenderanaJK, AkpoghenetaOJ, StrachanDL, HillZ, Ten AsbroekAH, et al. Mobile health (mHealth) approaches and lessons for increased performance and retention of community health workers in low- and middle-income countries: A review. Journal of Medical Internet Research2013;15(1):e17. [DOI: 10.2196/jmir.2130]PMC363630623353680 doi:10.2196/jmir.2130 PubMed
118. La MonicaE, OberstMT, MadeaAR, WolfRM. Development of a patient satisfaction scale. Research in Nursing & Health1986;9(1):43-50.10.1002/nur.47700901083634418 doi:10.1002/nur.4770090108 PubMed
119. LiberatiA, AltmanDG, TetzlaffJ, MulrowC, GøtzschePC, IoannidisJP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLOS Medicine2009;6(7):e1000100.10.1371/journal.pmed.1000100PMC270701019621070 doi:10.1371/journal.pmed.1000100 PubMed
120. LiddyC, DrosinisP, KeelyE. Electronic consultation systems: worldwide prevalence and their impact on patient care - a systematic review. Family Practice2016;33(3):274-85. [DOI: 10.1093/fampra/cmw024]27075028 doi:10.1093/fampra/cmw024 PubMed
121. LiddyC, MorozI, MihanA, NawarN, KeelyE. A systematic review of asynchronous, provider-to-provider, electronic consultation services to improve access to specialty care available worldwide. Telemedicine Journal and e-Health2019;25(3):184-98. [DOI: 10.1089/tmj.2018.0005]29927711 doi:10.1089/tmj.2018.0005 PubMed
122. MantelN, HaenszelW. Statistical aspects of the analysis of data from retrospective studies of disease. Journal of the National Cancer Institute1959;22(4):719-48.13655060 PubMed
123. MarsM. Building the capacity to build capacity in e-Health in Sub-Saharan Africa: the KwaZulu-Natal Experience. Telemedicine Journal and e-Health2012;18(1):32-7.10.1089/tmj.2011.0146PMC327004822150714 doi:10.1089/tmj.2011.0146 PubMed
124. McDowellI. Measuring Health: A Guide to Rating Scales and Questionnaires. 3rd edition. New York (NY): Oxford University Press, 2006.
125. Medicare Payment Advisory Commission (USA). Telehealth Services and the Medicare Program. Report to the Congress: Medicare and the Health Care Delivery System. www.medpac.gov/docs/default-source/reports/chapter-8-telehealth-services-and-the-medicare-program-june-2016-report-.pdf?sfvrsn=0 (accessed prior to 20 December 2017)2016:227-60.
126. MobasheriMH, KingD, JohnstonM, GautamaS, PurkayasthaS, DarziA. The ownership and clinical use of smartphones by doctors and nurses in the UK: A multicentre survey study. BMJ Innovations2015;1:174-81.
127. MorrisNS, MacLeanCD, ChewLD, LittenbergB. The Single Item Literacy Screener: Evaluation of a brief instrument to identify limited reading ability. BMC Family Practice2006;7:21. [DOI: 10.1186/1471-2296-7-21]PMC143590216563164 doi:10.1186/1471-2296-7-21 PubMed
128. NHS Commissioning Assembly. Technology Enabled Care Services: Resource for Commissioners, 2015. www.england.nhs.uk/wp-content/uploads/2014/12/TECS_FinalDraft_0901.pdf (accessed prior to 20 December 2017).
129. OdendaalWA, Anstey WatkinsJ, LeonN, GoudgeJ, GriffithsF, TomlinsonM, et al. Health workers’ perceptions and experiences of using mHealth technologies to deliver primary healthcare services: a qualitative evidence synthesis. Cochrane Database of Systematic Reviews2020, Issue 3. Art. No: CD011942. [DOI: 10.1002/14651858.CD011942.pub2]PMC709808232216074 doi:10.1002/14651858.CD011942.pub2 PubMed
130. AlkireS, RoblesG. Global Dimensional Povery Index 2017. Oxford Poverty & Human Development Initiative, ODID, 2017. Available from www.ophi.org.uk/wp-content/uploads/B47_Global_MPI_2017.pdf (accessed 13 October 2017).
131. Review Manager (RevMan) Version 5.3. Copenhagen: The Nordic Cochrane Centre, The Cochrane Collaboration, 2014.
132. SchünemannHJ, OxmanAD, HigginsJP, VistGE, GlasziouP, AklE, et al, on behalf of the Cochrane GRADEing Methods Group and the Cochrane Statistical Methods Group. Chapter 11: Completing ‘Summary of findings’ tables and grading the confidence in or quality of the evidence. In: Higgins JP, Churchill R, Chandler J, Cumpston MS, editor(s). Cochrane Handbook for Systematic Reviews of Interventions version 5.2.0 (updated June 2017). The Cochrane Collaboration, 2017. Available from www.training.cochrane.org/handbook.
133. Scottish Centre for Telehealth & Telecare. Programmes. sctt.org.uk/programmes (accessed 13 October 2017).
134. SteinhubiSR, MuseED, TopolEJ. Can mobile health technologies transform health care?JAMA2013;310(22):2395-6.10.1001/jama.2013.28107824158428 doi:10.1001/jama.2013.281078 PubMed
135. SterneJA, SuttonAJ, IoannidisJP, TerrinN, JonesDR, LauJ, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. BMJ2011;343:d4002. [DOI: 10.1136/bmj.d4002]21784880 doi:10.1136/bmj.d4002 PubMed
136. WallaceBC, Noel-StorrA, MarshallIJ, CohenAM, SmalheiserNR, ThomasJ. Identifying reports of randomized controlled trials (RCTs) via a hybrid machine learning and crowdsourcing approach. Journal of the American Medical Informatics Association2017;24(6):1165-8.10.1093/jamia/ocx053PMC597562328541493 doi:10.1093/jamia/ocx053 PubMed
137. World Health Organization Global Observatory for eHealth. Global Observatory for eHealth series - Volume 2. Telemedicine: Opportunities and developments in Member States. January 2011. www.who.int/goe/publications/ehealth_series_vol2/en/ (accessed 31 August 2017).
138. World Health Organization. Global Observatory for eHealth. Global diffusion of eHealth: Making universal health coverage achievable. Report of the third global survey on eHealth. 2016. www.who.int/goe/publications/global_diffusion/en/ (accessed 31 August 2017).
139. World Health Organization. Classification of Digital Health Interventions v1.0. WHO/RHR/19.06. www.who.int/reproductivehealth/publications/mhealth/classification-digital-health-interventions/en/ (accessed 24 April 2020).
140. World Health Organization. WHO Guideline: Recommendations on Digital interventions for Health System Strengthening. www.ncbi.nlm.nih.gov/books/NBK541902/pdf/Bookshelf_NBK541902.pdf 2019 (accessed 24 April 2020).31162915 PubMed
141. WuQ, ZhaoL, Ye X-C. Shortage of healthcare professionals in China. BMJ2016;354:i4860.10.1136/bmj.i486027659864 doi:10.1136/bmj.i4860 PubMed
142. Gonçalves-BradleyDC, BuckleyBS, FønhusMS, GlentonC, HenschkeN, LewinS, et al. Mobile-based technologies to support healthcare provider to healthcare provider communication and management of care. Cochrane Database of Systematic Reviews2018, Issue 1. Art. No: CD012927. [DOI: 10.1002/14651858.CD012927]PMC743739232813281 doi:10.1002/14651858.CD012927 PubMed