Clinical, service and technological evidence selection

Evidence search strategies and study selection

The search strategies are presented in Appendix 1 and Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) diagram for the evidence selection is presented in Figure 2, Appendix 2.

FIGURE 1 General structure of model. Square = decision node; circle = chance node; triangle = terminal node. D+ = prevalence (i.e. probability of a suspected fracture being a true fracture); D− = 1 – D+; T1+ = conditional probability of a positive result from first review of X-ray (in branch shown this is sensitivity); T2+ = conditional probability of a positive result from second review of X-ray. Costs and QALYs are assigned to terminal nodes. Image drawn with SilverDecisions (silverdecisions.pl). (Figure not reproduced here; see the original article.)

Searches were carried out in late June and early July 2024. The search strategies used relevant search terms for AI, X-ray and different fracture types; each of these subjects comprised a combination of indexed keywords [e.g. Medical Subject Headings (MeSH)] and free-text terms appearing in the titles and/or abstracts of database records. Searches were translated and adapted according to the configuration of each database. No date, language or publication status (published, unpublished, in-press and in-progress) limits were applied. Searches for clinical and service outcomes and cost-effectiveness were combined and carried out in one search strategy.

Following deduplication, a total of 1341 records of potentially relevant evidence on clinical and/or cost-effectiveness were retrieved. Databases searched were MEDical Literature Analysis and Retrieval System (MEDLINE) (including MEDLINE in Process), Excerpta Medica dataBASE (EMBASE), Cochrane, Web of Science, Cost-effectiveness Analysis (CEA) Registry and Health Economics Research Centre (HERC). Additional trial registries searched were Clinicaltrials.gov (National Library of Medicine) and International Clinical Trials Registry Platform (World Health Organization). The websites of the individual companies were searched; NICE and Scottish Intercollegiate Guidelines Network websites were searched for related guidelines; Manufacturer and User Facility Device Experience and Medicines and Healthcare products Regulatory Agency were searched for adverse events data. In addition, we scanned the reference lists for the Kuo et al.10 and Pauling et al.11 systematic reviews.

During study screening, the EAG identified several studies where the technology evaluated was unclear or where it was unclear whether the technology was evaluated as a stand-alone technology or with the interpretation of a clinician. In these cases, the EAG sought advice from three of the companies included in this assessment (Gleamer, Milvue and AZmed) where they were stated to have sponsored the study and/or where staff from their companies were listed as authors. The EAG did not receive a response from Milvue or AZmed during its assessment, and therefore the studies queried were not included in the review. Following a response from Gleamer, one study was included in the assessment and two studies were excluded. Additional queries about included studies were also sent to companies (see Chapter 7) for details, which resulted in the merging of two studies: Radiobotics12 and Bonde et al., Radiobotics, 3 July 2024. The former of these was a published document with little reported data (though following a request from the EAG the company, Radiobotics, provided additional data), and the latter was a full manuscript in preparation that reported data for a subgroup of participants in one of the included countries. A full list of exclusions is provided in Appendix 3, Tables 48 and 49.

Included and excluded studies

A total of 1343 titles and abstracts were screened, 209 full-text publications were reviewed, and 16 studies (17 documents) met the review inclusion criteria. The included studies are summarised in Table 2.

One study included in the review13 conducted a head-to-head comparison of assisted reading using three technologies: BoneView, Rayvolve and TechCare Alert (please note that the study referred to TechCare Alert as ‘SmartUrgences’, which is an umbrella name for AI technologies developed by the same manufacturer. For pragmatic purposes, the EAG assumed that these were the same technology). The majority of other studies included in the review evaluated readings assisted with either BoneView or RBfracture.

A breakdown of the number of studies evaluating each technology is as follows:

• BoneView: nine studies, including two non-comparative studies where BoneView assisted readings were assessed against the reference standard (Cohen,14 Meetschen19), one head-to-head comparison (Bousson13) and five studies that assessed both BoneView-assisted and unassisted readings (Canoni-Meynet,15 Dell-Aria,16 Duron,17 Guermazi,18 Nguyen,20 Oppenheimer21).
• RBfracture: five comparative studies (Bachmann,23 Jørgensen,24 Radiobotics12 (also Bonde et al., Radiobotics, 3 July 2024), Ruitenbeek,27 Yogendra et al., Radiobotics, 3 July 2024).
• Rayvolve: one comparative study (Fu22) and one head-to-head comparison (Bousson13).
• TechCare Alert: one comparative study (Suite26) and one head-to-head comparison (Bousson13).
• qMSK: zero studies.

Study design and diagnostic tests

Details about the methodological approach used by the included studies are provided in Table 3.

Six studies13–16,21,25 included consecutive cases presenting to participating centres during the study period and one study22 included a random selection of cases from a database of patients who presented with a suspected fracture. The remaining nine studies (Yogendra et al., Radiobotics, 3 July 2024)12,17–20,23,24,26 used a case-control design to stratify inclusion towards a set fracture prevalence rate (typically 50%, with additional requirements, such as spread of specific fracture types and age groups). Consecutive and random sampling study designs are generally more robust for diagnostic evaluation, as the study samples more closely represent the prevalence of the target condition that would be seen in clinical practice, meaning that prevalence data and positive predictive value (PPV) and negative predictive value (NPV) outcomes (each affected by the prevalence rate) will be more reliable. However, as the diagnostic accuracy of X-ray varies according to the fracture location, even consecutive and random sampling designs may be limited if they do not include a mix of fracture types that is representative of clinical practice (meaning that sensitivity and specificity data may not be generalisable). This issue is discussed further in Populations and reported prevalence.

The vast majority of studies used a retrospective study design, with only two studies16,21 using a prospective design. Retrospective designs are unlikely to be representative of clinical practice for several reasons, including that readers have knowledge that their judgement will not impact upon patient care, readers are less likely to have the same access to support from colleagues, and readers may not dedicate the same length of time and consideration to reading X-rays within these circumstances compared to clinical practice.

None of the included studies were set in the UK, though the majority (12 studies) included sites in Europe, 5 studies included sites in USA, and 1 study was set in Asia (Singapore). Based on the information reported in the included studies, it was not clear to the EAG how applicable the study findings were to the decision problem for the assessment; that is the target emergency settings of ED, MIU and UTC, and the extent to which readers of index tests and the reference standard were comparable to staff in the target settings. This was a major limitation of the evidence base, since – as noted in the review protocol2 – the diagnostic accuracy of the index test and technologies would be expected to vary according to both case mix and reader experience. Eleven of the included studies (Yogendra et al., Radiobotics, 3 July 2024)13–16,19,21,24–26 were described to be set within a hospital and/or trauma centre. However, the EAG was uncertain to what extent the care pathway and treating clinicians would vary between hospital settings in included studies as compared to the UK. Of the other studies, two studies17,22 were described as being set in medical settings and one study23 was described as being based within a virtual centre. Two studies18,20 were described as using data from unspecified settings with the USA.

Many of the included studies reported multiple analyses for the diagnostic accuracy of unassisted and unassisted readers according to different staff grades (in addition to other relevant subgroups, such as age group and fracture location). As it was not possible to determine which analyses were most relevant to NHS staff who would be expected to use the technology in clinical practice, the EAG instead sought to categorise available data according to the level of experience as described by the publications: less experienced staff; highly experienced staff, and mixed or unclear levels of experience. The EAG noted that these categorisations were based on highly limited information and there is a high risk of error. Further information about this is provided in Results from the evidence base and evidence synthesis. Very few studies included staff that appeared comparable to those who would typically work within urgent care settings in the NHS, such as emergency care doctors or specialist trauma nurses.

Only three of the included studies specified the version of the technology under evaluation: in these studies, it was stated that Bousson evaluated BoneView version 1.0.2. and TechCare Alert version 1.7 (no version reported for Rayvolve), Radiobotics evaluated RBfracture version 1.0 and Yogendra et al., Radiobotics, 3 July 2024 evaluated RBfracture version 2.0. Descriptions of the output of the evaluated technologies was typically poor across studies, and therefore the EAG did not feel confident in differentiating between studies on the basis of features mentioned or not mentioned in the publications. For example, two studies17,21 evaluating BoneView reported that the technology provided readers with an indication of confidence in the result, either as a note on the result and/or as an altered boundary box around the area of interest. One study (Yogendra et al., Radiobotics, 3 July 2024) that evaluated RBfracture also indicated that a confidence rating was provided with the result. However, the EAG considered it plausible that other studies included in the assessment also evaluated versions of the technology that provided a rating of confidence with results, and therefore did not use reporting of this feature to draw a comparison across studies. Similarly, three studies mentioned that staff received training in the AI prior to the study12,15,23 with varying levels of training ranging from months (Canoni-Meynet15) to written instructions and five training cases only (Bachmann23). The EAG considered it plausible that all studies would have trained readers in using the technology prior to the study, and that training was simply not described for most studies. The EAG therefore did also not use training requirements as a factor for comparing findings.

Where the same readers read the same exams assisted and unassisted, the washout period between assisted and unassisted readings ranged between no washout to 3 months (two studies20,21 had no washout, five studies15,18,22,23,25 had 1-month washout, one study13 had 2-month washout, and one study16 had 3-month washout). The EAG considered that a washout of 1 month or longer was sufficient to ensure that the reading clinicians did not recall their previous responses to an X-ray, or at least would do so in very few cases. The EAG considered that there was a risk of bias associated with no washout period, as readers may be guided by their previous responses using the alternative method. Where washout was not reported (NR), the EAG considered it more likely that no washout period was used, though of course this was not clear.

One of the studies14 used an index test that was considered not to be representative of the likely use of the technology: in this study to detect wrist fractures, rather than readers using the technology to reach a decision on diagnosis, the stand-alone results from the AI results and results of the original radiology report were artificially combined: an observation was considered positive when it was detected by either the AI or reported on initial radiology report, regardless of the other’s group result. The EAG considered this study to be at a risk of bias.

The reference standard used in the included studies generally included a decision from a senior radiologist, though there was some variation in the information that readers had to make their diagnosis (e.g. clinical information and medical notes, and access to further imaging). Where the reference standard was based on limited access to information about the patient and injury (as would be available within routine clinical practice), the EAG considered there to be an increased risk of incorrect judgements. Where a reference standard is determined to be imperfect, this affects the reliability of all the study results. The reference standard used in three studies (Canoni-Meynet15 Bousson13 and Yogendra28) included the results of the AI technology, and in a further study, it was unclear whether this was the case (Oppenheimer21). The inclusion of these studies was a deviation from the review protocol2 (as described in Decision problem) and was also associated with the potential for bias in the results, as it created closer alignment between the results of the AI-assisted assessment and the reference standard. The EAG considered that the evidence from these studies should be interpreted with caution.

On the basis of information about the study design used by the included studies, and (as described in Chapter 1), the EAG identified studies that provided the best quality evidence available within the evidence base and therefore could be used to inform the economic analysis. These decisions were based upon the following post hoc criteria: having a reference standard that did not include the AI reports, inclusion of results for both AI-assisted and -unassisted readers, relatively large sample sizes and peer-reviewed publication. Based upon these key criteria, the EAG considered the pivotal studies for each technology, to be as follows:

• BoneView: Duron17 (for adults) and Nguyen20 (for children and young adults).
• RBfracture: Bachmann23 (adults and children).
• Rayvolve: two studies evaluated Rayvolve (Fu22 and Bousson13). Although limited by a relatively small sample size, Fu avoided the limitations noted above and was therefore considered by the EAG to be the pivotal study for this technology.
• TechCare Alert: two studies evaluated TechCare Alert (Suite26 and Bousson13), both of which were associated with limitations as described above. Suite was also not presented in a peer-reviewed publication. Evidence for this technology was therefore of poorer quality than the studies listed above.

The EAG emphasises that all of these studies are associated with their own quality limitations that may undermine the reliability of the findings. In particular, none of the studies named above used a prospective design and may not be representative of clinical practice. Unfortunately, the two studies that included a prospective design were limited in other ways (see Quality appraisal of studies) and were considered to be less reliable than those selected. Specifically, in Oppenheimer21 it was unclear whether the reference standard included the results of the technology, and Dell-Aria16 was a relatively small study.

Participant selection was also a key consideration in the selection: studies where participants were selected to ensure sufficient numbers of fractures (i.e. case-control designs, where 50% of the sample were selected for the presence of a fracture) would not have a fracture prevalence rate representative of clinical practice (and may also be unlikely to have a representative fracture location mix). These studies were not used to determine fracture prevalence in economic analysis, and PPV and NPV estimates were considered to be unreliable. Three of the pivotal studies17,20,23 used a case-control design, but due to methodological issues with the other studies for these technologies, these studies remained the most robust for estimating sensitivity and specificity (prevalence, PPV and NPV were not extracted or considered from these studies). Both of the studies that evaluated TechCare Alert13,26 also used a case-control design, whereas the pivotal Rayvolve study22 used consecutive sampling.

Important subgroups in the review were different staff grade/type and specific fracture locations (hand/wrist, foot/ankle, hip, elbow fractures in children and Salter-Harris fractures). The majority of studies provided some relevant subgroup data, with four BoneView studies15,16,18,19 three RBfracture studies (Bonde et al., Radiobotics, 3 July 2024)12,23,28 and the Rayvolve22 and TechCare Alert studies26 all reporting some diagnostic test accuracy (DTA) data for different staff types or grades and five BoneView studies15,17,18,20,21 and the head-to-head study13 reporting some DTA data for different fracture locations. Oppenheimer21 and Bachmann23 each reported subgroup data in children and young people.

Populations and reported prevalence

Demographics about the study sample used in the included studies, including prevalence rates of any and subgroup fracture types, are shown in Table 4.

As noted in Study design and diagnostic tests, the EAG considered that studies using a case-control design would not provide prevalence data that were representative of clinical practice. The data from these studies should therefore be considered to only represent the case mix in the included studies and were not used as an estimate of fracture prevalence in the economic analysis. Studies with more reliable prevalence data (i.e. studies with consecutive and random sampling) are highlighted in bold. Studies typically only included adults, or the vast majority of the sample were adults (see Table 2). There were five studies13,15,19,21,23 that included a mix of adults, children and young people, of which two studies21,23 reported subgroup data specifically in children and young people. Two studies were conducted only in children and young people (Yogendra et al., Radiobotics, 3 July 2024).20

No studies reported frailty measures for its participants. Two studies12,24 reported that the mean age of the sample was > 70 years. One study16 conducted with participants with a median age of 39 years (no range reported) who had all experienced a low velocity trauma, and one study reported that 83.0% of injuries were due to falls. The EAG considered that all of these indicators may be correlated with frailty, but nevertheless would not provide a meaningful representation of outcomes in people with frailty.

The EAG sought additional information from studies about the health of participants, such as the number of participants with diseases that affect bone health (e.g. osteoporotic disease). No studies reported this information.

Eight studies14–20,23 reported that a minority of participants had multiple fractures; where rates were reported, these ranged from 3.3% to 26% of the study sample. None of the studies excluded suspected dislocations and effusions, although only one study26 stated that the study sample included these injuries. Three studies12,16,17 explicitly excluded ‘obvious’ fractures, such as open fractures, displaced, and multifragmented and those caused from polytrauma, while other studies neither included nor excluded these. Within the EVA, the EAG did not extract information of the full range of fracture types included in the study samples from which to consider the case mix in which the technologies were evaluated. As noted in Study design and diagnostic tests, the EAG considered that studies using a case-control design would not provide prevalence data that were representative of clinical practice. The data from these studies should therefore be considered to only represent the case mix in the included studies and were not used as an estimate of fracture prevalence in the economic analysis. The EAG noted that the most methodologically robust study for the estimation of prevalence was likely Cohen14 but this study focused only on wrist fractures. Other studies that were reasonably robust for estimating prevalence were Dell-Aria16 and Fu22 (albeit both limited due to relatively small sample sizes), and Ruitenbeek25 (only presented in an abstract, and therefore providing very limited data). Among the studies that used consecutive and random sampling, rates of subgroup fracture types were rarely reported. Where these were available, these varied between studies, suggesting that the study samples included different underlying populations. As diagnostic accuracy for detecting fractures on X-rays varies according to fracture location, variation in case mix across studies will likely influence the findings.

Outcomes

The outcomes reported by, or calculable from, the included studies are shown in Table 5.

Sensitivity, specificity and contingency tables were reported or calculable for all studies, although some data were missing for specificity and contingency tables for subgroup analyses (staff grade and fracture types) in two studies.15,21 PPV and NPV were either NR or were not extracted for case-control studies. Additional outcomes reported by a small number of studies were the prevalence of fractures, as assessed using the reference standard (see Chapter 1), and the reading time for radiographs. The vast majority of outcomes listed on the NICE scope1 were NR in any of the included studies (see Chapter 9).

Diagnostic accuracy that was reported per patient (i.e. a binary decision about whether a patient had a fracture) was typically reported across studies and prioritised for inclusion in the review. If these data were not provided, available per-exam or per-fracture data were extracted and these instances are highlighted in the results. The decision to prioritise per-patient data was made primarily because the majority of studies report data in this way and because these data were most relevant to the economic analysis. The EAG noted that the per patient approach allows for the assessment of one injury only, while the per-fracture approach accounts for accuracy across multiple fractures in the same person, and thus means that there will be multiple reports for the same participant. The per-exam approach is similar to the per-patient approach and usually reports on one injury per patient but may plausibly detect more than one injury in the same location. As noted in Populations and reported prevalence, the samples of eight of the studies included people with multiple fractures, all of which reported per-patient data, which were extracted for the assessment. These data, therefore, do not fully capture the rates of fractures in the study samples and may overestimate the sensitivity of the technologies [since a true positive (TP) result may be acquired even if all fractures in the same person are not identified].

Quality appraisal of studies

As this was an EVA, no formal quality assessment of the included studies was conducted. Quality considerations that were considered to represent a potential source of bias have been discussed throughout the previous sections.

An overview of the way quality considerations influenced the interpretation of diagnostic evidence and the selection of evidence to inform the economic analysis are shown in Table 6. Coloured ratings are provided for ease of reference, though the EAG emphasise that these ratings are relative to each other and none of the included evidence was considered to be robust for diagnostic outcomes.