Results from the evidence base and evidence synthesis
Diagnostic accuracy
The diagnostic accuracy of the technologies as reported by the included studies and/or calculated by the EAG is shown in Table 7. Data were calculated where these were NR in publications. In some cases, calculated data varied slightly from that reported in the publication, which the EAG assumed was due to rounding errors, alternative methods for handling missing data, or reporting errors. The EAG selected the most reliable data for inclusion in the report tables and the evidence synthesis, meaning that some data [in all cases, rates of TP, FP, true negative (TN), false negative (FN)] differed from those reported in the publications. As noted in Outcomes, diagnostic data were calculated per patient (i.e. one result per patient, meaning that the analysis did not account for multiple fractures in the same person), unless otherwise indicated.
| Author. Target fractures | Prev (reference standard) | Less experienced readers assisted | Less experienced readers unassisted | Mixed or unclear staff level assisted | Mixed or unclear staff level unassisted | Senior and highly experienced staff assisted | Senior and highly experienced staff unassisted |
|---|---|---|---|---|---|---|---|
| BoneView | |||||||
| Bousson13Age range NR; likely all or mostly adultsHand, wrist, arm, elbow, shoulder, pelvis, hip, leg, knee, ankle, foot | 326/1500 (26.9%) | . | . | Radiology residents (4 years of residency)Sensitivity 91.3 (88.2–93.6); specificity 90.5 (89.1–92.3); PPV NR; NPV NRTP 298FP 112TN 1062FN 28 | . | . | . |
| Canoni-Meynet15Age 0.25–99 years; 80.2% were adultsAll fractures excluding skull and face | 188/500 (37.6%) | Third year radiology resident.Sensitivity 89 (83–93); specificity 93 (89–95); PPV 88 (83−92); NPV 93 (90–96)TP 167FP 23TN 189FN 21 | Third year radiology resident.Sensitivity 68 (60–74); specificity 96 (93–98); PPV 91 (87–96); NPV 83 (79–87)TP 127FP 12TN 300FN 61 | . | . | Senior radiologistSensitivity 88 (83–93); specificity 99 (98–100); PPV: 98 (97–100); NPV 93 (91–96)TP 166FP 2TN 310FN 22 | Senior radiologist.Sensitivity 70 (63–77); specificity 96 (94–98); PPV 92 (88–97); NPV 84 (80–88)TP 132FP 11TN 301FN 56 |
| Dell-Aria16Age range NR; mean age 39 yearsLow velocity trauma to upper or lower limbs (including shoulder and hip) | 54/101 (53.9%)Injuries that required additional imaging: 32/51 (62.7%) | AI-assisted radiologist with < 5 years’ experienceInjuries that did not require other imaging. Sensitivity 77.27; specificity 88.29; PPV 85; NPV 83.33Injuries that did require additional imagingSensitivity 56.25; specificity 78.95; PPV 81.82; NPV 51.72TP 18FP 4TN 15FN 14 | Unassisted radiologist with < 5 years’ experienceInjuries that did not require other imaging. Sensitivity 50.0; specificity 82.14; PPV 68.75; NPV 67.65Injuries that did require additional imagingSensitivity 31.25; specificity 89.47; PPV 83.33; NPV 43.59TP 10FP 2TN 17FN 22 | . | . | AI-assisted radiologist with > 15 years’ experienceInjuries that did not require other imaging. Sensitivity 95.45; specificity 89.29; PPV 87.5; NPV 96.15Injuries that did require additional imaging.Sensitivity 81.25; specificity 89.47; PPV 98.86; NPV 73.91TP 26FP 2TN 17FN 6 | Unassisted radiologist with > 15 years’ experienceInjuries that did not require other imaging. Sensitivity 86.36; specificity 89.29; PPV 86.36; NPV 89.29Injuries that did require additional imaging.Sensitivity 56.25; specificity; 89.47; PPV 90; NPV 54.84TP 18FP 2TN 17FN 14 |
| Duron17Age range 18–100 years; mean 57Shoulder, arm, hand, pelvis, leg, foot | 300/600 (50%) | Assisted emergency physiciansSensitivity 74.3 (SD 6.6); specificity 96.6 (1.9)TP 223FP 10TN 290FN 77 | Unassisted emergency physiciansSensitivity 61.3 (SD 9.3); specificity 90.6 (5.8)TP 184FP 28TN 272FN 116 | AI-assisted six radiologists and six emergency physicians, including residents and expertsSensitivity 79.4% (SD: 7.4); specificity 93.6% (4.6); PPV NR; NPV NRTP 238FP 19TN 281FN 62 | Unassisted six radiologists and six emergency physicians, including residents and expertsSensitivity 70.8% (SD: 12.5); specificity 89.5% (6.5); PPV NR; NPV NRTP 212FP 31TN 269FN 88 | . | . |
| Meetschen19Age range 1–95 years; mean 40.7.Hand, wrist, arm, elbow, shoulder, scapula, clavicle, ribs, spine, pelvis, hip joints, legs, knees, ankles and feet | 100/200 (50%) | . | . | Radiology residents (4.5 to 24.5 months of experience).Sensitivity 77 (72–82); specificity 79 (73–84); PPV 81% (76–86); NPV 75% (69–80)TP 77FP 21TN 79FN 23 | A radiology resident without AI.Sensitivity 58 (52–64); specificity 77 (71–81); PPV 74 (67–79); NPV 62 (56–67)TP 58FP 23TN 77FN 42 | ||
| Nguyen20Children and young people, age range 2–21 years; mean 10.8Appendicular skeleton | 150/300 (50%) | . | . | AI-assisted, eight readers: 5 radiology residents (between the second and fourth year of residency) and 3 expert paediatric radiologists (at least 7 years of experience, including > 3 years specialising in paediatric radiology)Sensitivity 82.67 (75.65–88.36); specificity 90.33 (84.43–94.55); PPV NR; NPV NRTP 124FP 15TN 135FN 26 | Unassisted eight readers: 5 radiology residents (between the second and fourth year of residency) and 3 expert paediatric radiologists (at least 7 years of experience, including > 3 years specialising in paediatric radiology)Sensitivity 73.17 (65.33–80.07); specificity 89.58 (83.55–93.97); PPV NR; NPV NRTP 110FP 16TN 134FN 40 | . | . |
| Oppenheimer21,aAge range 2–100 years; mean 61.4.All excluding cervical spine, skull and face | 367/1163 (31.56%) | . | . | Resident radiologist with AISensitivity 91.28 (91.25–91.31); specificity 97.36 (97.35–97.37); PPV 94.10 (94.08–94.12); NPV 96.03 (96.02–96.04)TP 335FP 21TN 775FN 32 | Resident radiologists without AISensitivity 84.74 (84.70–84.78); specificity 97.11 (97.10–97.12); PPV 93.11 (98.08–93.14); NPV 93.24 (93.22–93.26)TP 311FP 23TN 773FN 56 | . | . |
| Rayvolve | |||||||
| Bousson13 Age range NR; likely all or mostly adultsHand, wrist, arm, elbow, shoulder, pelvis, hip, leg, knee, ankle, foot | 326/1500 (26.9%) | AI-assisted radiology residents (4 years of residency)Sensitivity 92.6 (90.1–94.6); specificity 70.4 (68.1–73); PPV NR; NPV NRTP 302FP 348TN 826FN 24 | |||||
| Fu22,bAdults aged ≥ 21 years. Ankle, clavicle, elbow, forearm, humerus, hip, knee, pelvis, shoulder, tibia/fibula, wrist, hand, foot | 587/2626 (22.4%) | Emergency physician (assisted).Sensitivity 0.938 (0.915–0.955); specificity 0.853 (0.828–0.875); PPV NR; NPV NRTP 551FP 300TN 1739FN 36 | Emergency physician (unassisted).Sensitivity 0.792 (0.757–0.824); specificity 0.852 (0.828–0.874); PPV NR; NPV NRTP 465FP 302TN 1737FN 122 | Eight each of emergency physicians, non-MSK radiologists, and MSK radiologistsSensitivity 0.955 (0.944–0.964); specificity 0.831 (0.817–0.845); PPV NR; NPV NRTP 508FP 355TN 1684FN 79 | Eight each of emergency physicians, non-MSK radiologists, and MSK radiologistsSensitivity 0.865 (0.848–0.881); specificity 0.826 (0.812–0.840); PPV NR; NPV NRTP 561FP 345TN 1694FN 26 | . | . |
| RBfracture | |||||||
| Bachmann23Age range NR; 70.7% adultsAppendicular skeleton | 164/334 (49.1%) | Assisted A&E trainees.Sensitivity 0.83 (0.81–0.86); specificity 0.90 (0.87–0.92); PPV NR; NPV NRTP 136FP 17TN 153FN 28Trauma-care nurses Sensitivity 0.70 (0.65–0.75); specificity 0.67 (0.62–0.72)TP 115FP 56TN 114FN 49 | Unassisted A&E trainees.Sensitivity 0.74 (0.71–0.78); specificity 0.87 (0.84–0.89); PPV NR; NPV NRTP 121FP 22TN 148FN 43Trauma-care nurses Sensitivity 0.58 (0.53–0.64) specificity 0.60 (0.55–0.65)TP 95FP 68TN 102FN 69 | 2 advanced trauma care nurses, 3 diagnostic radiographers, 4 A&E trainees, 3 orthopaedic specialty registrars, 3 radiology specialty registrars. Sensitivity 0.80 (0.78–0.82); specificity 0.85 (0.84–0.87); PPV NR; NPV NRTP 131FP 25TN 145FN 33 | Unassisted 2 advanced trauma care nurses, 3 diagnostic radiographers, 4 A&E trainees, 3 orthopaedic specialty registrars, 3 radiology specialty registrars. Sensitivity 0.72 (0.70–0.73); specificity 0.81 (0.80–0.83); PPV NR; NPV NRTP 118FP 32TN 138FN 46 | . | . |
| Ruitenbeek25Age range 17–99 yearsAppendicular skeleton | 649/1500 (43.3%) | Assisted junior radiographer. | 649/1500 (43.3%) | Assisted junior radiographer | 649/1500 (43.3%) | . | . |
| Yogendra28Children aged 2–15 years/Appendicular skeleton | 145/500 (29%) | . | . | One senior resident or senior registrar in diagnostic radiology training (≥ 3 months’ training in paediatric radiology and had undergone a total of 3–4 years of formal residency training)Sensitivity 0.9; specificity 0.99; PPV 0.98; NPV0.96TP 131FP 4TN 351FN 14 | Two senior residents or senior registrars in diagnostic radiology training without AI (≥ 3 months’ training in paediatric radiology and had undergone a total of 3–4 years of formal residency training)Sensitivity staff 1 : 0.77, staff 2 : 0.77; specificity staff 1 : 0.97, staff 2 : 0.92; PPV staff 1 : 0.93, staff 2 : 0.8; NPV staff 1 : 0.91, staff 2 : 0.91TP 112cFP 20cTN 335cFN 33c | One associate/junior consultant plus AI (≥ 3 months’ training in paediatric radiology and had completed 5 years of local residency training subsequent to residency exit)Sensitivity 0.77; specificity 0.98; PPV 0.93; NPV0.91TP 112FP 43TN 312FN 33 | Two associate/junior consultants without AI (≥ 3 months’ training in paediatric radiology and had completed 5 years of local residency training subsequent to residency exit)Sensitivity staff 1 : 0.71, staff 2 : 0.59; specificity staff 1 : 0.98, staff 2 : 0.99; PPV staff 1 : 0.93, staff 2 : 0.96; NPV staff 1 : 0.89, staff 2 : 0.86TP 94cFP 5cTN 350cFN 51c |
| TechCare Alert | |||||||
| Bousson13Age range NR; likely all or mostly adultsHand, wrist, arm, elbow, shoulder, pelvis, hip, leg, knee, ankle, foot | 326/1500 (26.9%) | . | . | Assisted radiology residents (4 years of residency)Sensitivity 90.2 (87.2–92.8); specificity 92.5 (91.1–94); PPV NR; NPV NRTP 294FP 88TN 1086FN 32 | . | . | . |
| Suite26Adults aged 18–98 years; mean age 53.6Lower limbs, upper limbs, ribs | 253/620 (40.8%) | Four junior radiologists with AISensitivity 95; specificity 98; PPV 97; NPV 96TP 240FP 7TN 360FN 13 | Junior radiologist without AI.Sensitivity 92; specificity 97; PPV 96; NPV 94TP 233FP 11TN 356FN 20 | . | . | Senior radiologist with AI.Sensitivity 93; specificity 98; PPV 97; NPV 96TP 235FP 7TN 360FN 18 | Senior radiologist without AI.Sensitivity 93; specificity 98; PPV 97; NPV 96TP 235FP 7TN 360FN 18 |
Only three studies17,22,23 reported outcomes for readers based within emergency care settings, each evaluating a different technology: one study17 compared assisted and unassisted readings with BoneView as read by emergency physicians; one study22 reported assisted and unassisted readings with Rayvolve as read by emergency physicians; and one study23 reported assisted and unassisted readings with RBfracture as read by accident and emergency (A&E) trainees or trauma-care nurses. Most of the included studies reported multiple analyses according to the level of experience of clinicians reading the X-rays. As the protocol2 for this assessment included a comparison of diagnostic accuracy across staff reading experience, evidence reported for readers of varying levels of experience was extracted. The EAG noted that those with more seniority and expertise in reading X-rays, would not be expected to use the technology in clinical practice and that these data were presented for comparison purposes only. To aid interpretation of the findings across reader experience, the EAG grouped results according to the description of reader experience and seniority described in the publications. As all of the included studies were based outside of the UK, staff grades used in the publications had unclear relevance to target clinicians within NHS settings, which meant that some groupings were uncertain. Three staff groupings were created: readers described by studies as having less experience with reading X-rays, including staff specified as being based within emergency settings as well as those stated to have less experience in reading X-rays; senior and expert staff, intended to be consistent with a consultant radiologist and reporting radiographer grade; and a mixed and unclear grouping, where results were reported for groups of readers who varied widely in experience level and studies where reader descriptions did not fit easily within the less experienced and senior groups. The EAG noted that, in some studies, staff described as radiologists were nevertheless described as having less experience in reading X-rays, leading the EAG to assume that the term ‘radiologist’ may have a broader range of experience than would be expected within the NHS. In these cases, the EAG classified these as less experienced staff, though considered these groupings to be particularly uncertain, since descriptions in the publications may have been misleading.
Inspection of the results suggested that the groupings of studies according to reader experience were not particularly successful: in general, readers with greater seniority and expertise at reading X-rays were associated with more accurate diagnosis unassisted, though this was not universal across groupings. As this lacked face validity, the EAG assumed that the groupings according to reader experience were incorrect in some instances and so should be interpreted with caution when pooled (see Evidence synthesis).
Evidence from across the comparative studies generally suggested a trend for the technologies to improve sensitivity for diagnosing fractures in a mix of fracture types and across all reader groupings, but to have little or no impact on specificity. This trend was also present in those studies17,22,23 specific to readers based within emergency care settings. The comparability of outcomes between studies in emergency care settings was uncertain due to variation in the sensitivity of unassisted emergency physicians considered in both the Duron17 and Fu22 studies (sensitivity was reported as 61.3% in Duron and 79.2% in Fu), suggesting that staff experience or approach to decision-making varied between the studies. Nevertheless, the technologies improved sensitivity by a similar amount, with little or no change in specificity. Assistance with RBfracture in Bachmann23 improved sensitivity for trauma care nurses also by a similar amount, also with some benefit to specificity. The benefit to sensitivity for A&E trainees with RBfracture in the same study was smaller, with minimal benefit to specificity. However, in this study, unassisted A&E trainees had higher accuracy for detecting fractures unassisted than trauma care nurses and were no worse than either of the emergency physicians in Duron17 and Fu.22
Evidence from selected pivotal studies
The selection of pivotal studies for informing the economic analysis is described previously in this report. Nevertheless, the EAG cautions again that these studies were selected for being the most robust estimates within their design compared to other studies identified in the evidence base, but nevertheless still have limitations and should be interpreted with caution.
For BoneView, in a mixed group of readers, including radiologists and ED doctors, Duron17 reported sensitivity as 79.4% [standard deviation (SD) 7.4] when assisted by the technology and 70.8% (SD 12.5) when unassisted. Specificity was 93.6% (SD 4.6) assisted and 89.5% (SD 6.5) unassisted. All participants in this study were adults. Similarly, using readers who were radiology residents with variable levels of experience, Nguyen20 reported a higher sensitivity with AI assistance (82.67%, 95% CI 75.65 to 88.36) than without assistance (73.17%, 95% CI 65.33 to 80.07) and a higher AI-assisted specificity (90.33%, 95% CI 84.43 to 94.55) than without AI assistance (89.58%, 95% CI 83.55 to 93.97). Nguyen 2022 reported subgroup data for children and young adults, and these data were presented in Subgroup results (paediatric participants) alongside data for children from other studies.
For RBfracture the pivotal study of sensitivity and specificity estimates was Bachmann23 (albeit with the caveats described above). In this study, when readers were a mixed group of emergency care staff with a moderate level of experience, sensitivity was reported as higher with AI assistance (0.80, 95% CI 0.78 to 0.82) than without (0.72, 95% CI 0.70 to 0.73). Likewise, AI-assisted specificity was higher (0.85, 95% CI 0.84 to 0.87) than unassisted (0.81, 95% CI 0.80 to 0.83). Subgroup data were also available for children [see Subgroup results (paediatric participants)], and for more junior staff (see Table 7).
Similarly, the pivotal Rayvolve study,22 provided sensitivity and specificity data for a mixed group of readers [emergency physicians, non-MSK radiologists, and musculoskeletal (MSK) radiologists] and reported a higher sensitivity with AI assistance (0.955, 95% CI 0.944 to 0.964) than without assistance (0.865, 95% CI 0.848 to 0.881). In this study, specificity was similar with and without AI assistance (0.831, 95% CI 0.817 to 0.845 and 0.826, 95% CI 0.812 to 0.840 respectively). Subgroup data were available for emergency physicians (see Table 7). The TechCare Alert study26 provided data only for junior radiologists, and data were sparse and without confidence intervals (CIs) (see Table 7).
The EAG noted that although these sensitivity and specificity estimates may appear to differ across technologies, due to clinical and methodological heterogeneity between studies, and the overall paucity of robust data, it remains difficult to comment on whether any AI technology performed better than another. Although the head-to-head study13 provided sensitivity and specificity data across three of the included technologies, the EAG highlighted that these data were likely limited to a greater extent than the pivotal studies by the inclusion of the AI results in the reference standard. This study also did not include readers likely to be involved in emergency assessments of X-rays in UK clinical practice.
The pivotal studies for sensitivity and specificity estimates did not report other DTA data, due to study design features previously described. The EAG emphasised the need for caution when interpreting all DTA results (all studies are likely to be very limited), and in particular any prevalence, PPV and NPV estimates provided in the studies. When looking at the PPV and NPV data, the most robust data were from Cohen,14 but these estimates were only relevant to wrist fractures (see Table 7).
Positive predictive value and negative predictive value from other included studies
Other studies with potentially reasonable designs for estimating PPV and NPV were Dell-Aria16 for BoneView, Ruitenbeek25 for RB Fracture, and Fu22 for Rayvolve (noting that Ruitenbeek and Fu are limited by the use of retrospective study designs, and Dell-Aria by a relatively small sample size). Indeed, of these three studies, only Dell-Aria16 reported PPV and NPV data (see Table 7). The EAG calculated PPV and NPV for the other two studies but highlighted the potential lack of generalisability to clinical practice. For Fu22 for a mixed group of readers, the EAG calculated PPV was slightly higher with Rayvolve than without (61.92 and 58.86 respectively), as was NPV (98.49 and 95.52 respectively). Similar results were found for the junior readers, that is the EAG calculated PPV was slightly higher with Rayvolve than without (64.75 and 60.63 respectively), as was NPV (97.97 and 93.44 respectively). For Ruitenbeek25 for a mixed group of readers, the EAG calculated PPV was slightly higher with RBfracture than without (85.65 and 83.73 respectively), as was NPV (89.59 and 82.53 respectively). PPV and NPV were overall lower for the junior readers but were similarly higher with RBfracture than without (PPV 69.51 and 66.57 respectively; NPV 88.38 and 76.00 respectively).
Subgroup results (paediatric participants)
Four studies reported diagnostic accuracy data in a sample of children and young people: two studies (Yogendra et al., Radiobotics, 3 July 2024)20 were conducted only with children and young people, and are also included in the previous section, and two studies21,23 reported subgroup data for children and young people. These data were in a mixed fracture population only and were only available in mixed/unclear and senior/expert reader groupings. The data are summarised in Table 8.
| Author | Prevalence (reference standard) | Test | Mixed or unclear reader assisted | Mixed or unclear reader unassisted | Senior and expert staff assisted | Senior and expert staff unassisted |
|---|---|---|---|---|---|---|
| BoneView | ||||||
| Nguyen20Appendicular skeleton | 150/300 (50%) | BoneView | AI-assisted, eight readers: 5 radiology residents (between the second and fourth year of residency) and 3 expert paediatric radiologists (at least 7 years of experience, including > 3 years specialising in paediatric radiology)Sensitivity 82.67 (75.65–88.36); specificity 90.33 (84.43–94.55); PPV NR; NPV NRTP 124FP 15TN 135FN 26 | Unassisted eight readers: 5 radiology residents (between the second and fourth year of residency) and 3 expert paediatric radiologists (at least 7 years of experience, including > 3 years specialising in paediatric radiology)Sensitivity 73.17 (65.33–80.07); specificity 89.58 (83.55–93.97); PPV NR; NPV NRTP 110FP 16TN 134FN 40 | . | . |
| Oppenheimer21All fractures excluding cervical spine, skull and face | 6/31 (19.4%) | BoneView | Resident radiologist with AISensitivity 100%; specificity 92%TP 6FP 2TN 23FN 0 | Resident radiologist without AISensitivity 100%; specificity 92%TP 6FP 2TN 23FN 0 | . | . |
| RBfracture | ||||||
| Bachmann23Appendicular skeleton | 49/98 (50%)a | RBfracture | Two advanced trauma care nurses, 3 diagnostic radiographers, 4 A&E trainees, 3 orthopaedic specialty registrars, 3 radiology specialty registrars.Sensitivity 0.89 (0.87–0.92); specificity 0.80 (0.77–0.83)TP 44FP 10TN 39FN 5 | Two advanced trauma care nurses, 3 diagnostic radiographers, 4 A&E trainees, 3 orthopaedic specialty registrars, 3 radiology specialty registrarsSensitivity 0.78 (0.74–0.81); specificity 0.77 (0.74–0.80)TP 38FP 11TN 38FN 11 | . | . |
| Yogendra28Appendicular skeleton | 145/500 (29%) | RBfracture | One senior resident or senior registrar in diagnostic radiology (≥ 3 months’ training in paediatric radiology and had undergone a total of 3–4 years of formal residency training).Sensitivity 0.9; specificity 0.99; PPV 0.98; NPV0.96TP 131FP 4TN 351FN 14 | Two senior resident or senior registrar in diagnostic radiology training without AI (≥ 3-months training in paediatric radiology and had undergone a total of 3–4 years of formal residency training)Sensitivity staff 1 : 0.77, staff 2 : 0.77; specificity staff 1 : 0.97, staff 2 : 0.92; PPV staff 1 : 0.93, staff 2 : 0.8; NPV staff 1 : 0.91, staff 2 : 0.91TP 112FP 20TN 335FN 33 | One associate/junior consultant plus AI (≥ 3-months training in paediatric radiology and had completed 5 years of local residency training subsequent to residency exit)Sensitivity 0.77; specificity 0.98; PPV 0.93; NPV0.91TP 112FP 43TN 312FN 33 | Two associate/junior consultants without AI (≥ 3 months’ training in paediatric radiology and had completed 5 years of local residency training subsequent to residency exit)Sensitivity staff 1 : 0.71, staff 2 : 0.59; specificity staff 1 : 0.98, staff 2 : 0.99; PPV staff 1 : 0.93, staff 2 : 0.96; NPV staff 1 : 0.89, staff 2 : 0.86TP 94FP 5TN 350FN 51 |
One of the studies21 evaluating BoneView reported 100% sensitivity for both assisted and unassisted readers. In other studies, (Yogendra et al., Radiobotics, 3 July 2024)20,23 the use of assisted readings with BoneView and RBfracture improved sensitivity for detecting fractures in children and young people but made no difference to specificity. Sensitivity for assisted diagnosis in these studies was high but would nevertheless result in more than 10% of positive fractures being missed.
Subgroup results (fracture type)
The EAG identified diagnostic evidence separately for hand/wrist fractures, foot/ankle fractures, hip/pelvis fractures and Salter-Harris II fractures. These data are shown in Table 9.
| Author | Prevalence (reference standard) | Test | Less experienced readers assisted | Less experienced readers unassisted | Mixed or unclear reader assisted | Mixed or unclear reader unassisted | Senior and expert readers assisted | Senior and experts unassisted |
|---|---|---|---|---|---|---|---|---|
| Hand/wrist | ||||||||
| Bousson13 | 97/314 (30.9%) | BoneView | Radiology residents (4 years of residency). Sensitivity 91.5 (84.9–95.6); specificity 92.1 (89.2–95.8)TP 89FP 17TN 200FN 8 | |||||
| Canoni-Meynet15,a | 38/NR | BoneView | . | . | Radiologists assisted sensitivity: hand 89.5% (33.9/38) | Radiologists unassisted sensitivity: hand 68.4% (25.9/38) | . | . |
| Cohen14 | 247/637 (38.8%); | BoneView | . | . | . | . | Artificial combination of AI + initial radiology reportSensitivity 88%(84–92); specificity 92%(89–95); PPV 88% (84–92); NPV 93% (90–95)TP 218FP 29TN 361FN 29 | Initial radiology reportSensitivity 76 (70, 81); specificity 96 (94, 98); PPV 93 (90, 97); NPV87 (83–90)TP 189FP 14TN 376FN 58 |
| Duron17 | Hand only: 44/108 (40.7%) | BoneView | Six radiologists and six emergency physicians Sensitivity 80.2% (SD: 11.4); specificity 91.0% (SD 6.4)TP 35FP 6TN 58FN 9 | Six radiologists and six emergency physicians Sensitivity 59.6% (SD: 20.5); specificity 84.7% (SD 11.0)TP 26FP 10TN 54FN 18 | ||||
| Nguyen20 | 30/60 (50%) | BoneView | Average across 5 radiology residents and 3 expert paediatric radiologists.Sensitivity 87.08 (69.79–96.46); specificity 88.33 (71.34–97.1)TP 26FP 4TN 26FN 4 | Average across 5 radiology residents and 3 expert paediatric radiologistsSensitivity 68.75 (49.31–84.32); specificity 87.92 (85.44–89.48)TP 21FP 4TN 26FN 9 | ||||
| Oppenheimer21 | NR | BoneView | Resident radiologist. Sensitivity 95.65% | Resident radiologist Sensitivity 78.26% | ||||
| Bousson13 | 97/314 (30.9%) | Rayvolve | Radiology residents (4 years of residency). Sensitivity 97.8 (94.5–99); specificity 74.6 (70.1–80.9)TP 95FP 55TN 162FN 2 | . | ||||
| Bousson13 | 97/314 (30.9%) | TechCare Alert | Radiology residents (4 years of residency). Sensitivity 93.6 (88–96.3); specificity 91.7 (89–95.3)TP 91FP 18TN 199FN 6 | |||||
| Foot/ankle | ||||||||
| Bousson13 | Foot: 56/186 (30.1%), ankle 42/232 (18.1%) | BoneView | Radiology residents (4 years of residency). Sensitivity Foot 98.1% (94.4–98.3), Ankle 89.9% (79.4–95.4%)FootTP 55FP 27TN 103FN 1AnkleTP 38FP 13TN 177FN 4 | |||||
| Canoni-Meynet15,a | 39/NR | BoneView | . | . | Radiologists assisted sensitivity: foot 82.9% (32.3/39) | Radiologists unassisted sensitivity: Foot 57.3% (22.3/39) | . | . |
| Duron17 | Foot only 44/84 (52.3%) | BoneView | Six radiologists and six emergency physicians Sensitivity 86.9% (SD: 8.3); specificity 92.9% (SD 5.8) | Six radiologists and six emergency physicians Sensitivity 71.8% (SD: 13.6); specificity 88.0% (SD 9.9) | ||||
| Nguyen20 | 30/60 (50%) | BoneView | Average across 5 radiology residents and 3 expert paediatric radiologists. Sensitivity 70.83 (51.47–85.89); specificity 86.25 (68.77–96.02)TP 21FP 4TN 26FN 9 | Average across 5 radiology residents and 3 expert paediatric radiologists. Sensitivity 70.83 (51.47–85.89); specificityTP 16FP 4TN 26FN 14 | ||||
| Oppenheimer21 | NR | BoneView | Resident radiologist. Sensitivity 88.57% | Resident radiologist. Sensitivity 82.86% | ||||
| Bousson13 | Foot: 56/186 (30.1%), ankle 42/232 (18.1%) | Rayvolve | Radiology residents (4 years of residency). Sensitivity Foot 90.8% (83.0–95.0), Ankle 92.1% (82.2–96.5)FootTP 51FP 49TN 81FN 5AnkleTP 39FP 53TN 137FN 3 | |||||
| Bousson13 | Foot: 56/186 (30.1%), ankle 42/232 (18.1%) | Smarturgences | Radiology residents (4 years of residency). Sensitivity Foot 85.4% (73.1–90.2), Ankle 89.9% (76.0–95.5)FootTP 48FP 12TN 118FN 8AnkleTP 38FP 15TN 175FN 4 | |||||
| Hip | ||||||||
| Oppenheimer21 | NR | BoneView | Resident radiologist. Sensitivity 93.22% | Resident radiologist. Sensitivity 93.22% | ||||
| Jørgensen24 | 107/214 (50%) | RBfracture | . | . | Two radiographers, two medical interns and two consultants.Sensitivity 0.96 (CI: 0.95 to 0.98); specificity 0.86 (CI: 0.83 to 0.89); PPV NR; NPV NRTP 103FP 15TN 92FN 4 | Two radiographers, two medical interns and two consultants.Sensitivity 0.93 (CI: 0.91 to 0.95); specificity 0.87 (CI: 0.85 to 0.90); PPV NR; NPV NRTP 100FP 14TN 93FN 7 | . | . |
| Radiobotics;12 Bonde; confidential study reportNon-obvious proximal femur fractures, including the femoral neck and head | Less experienced readers: 126/237 (53%); Mixed or unclear readers 156/312 (50%) | RBfracture | Readers (with < 2 years’ experience) with AISensitivity 0.93 (0.89–0.97); specificity 0.94 (0.86–0.92); PPV NR; NPV NRTP 111FP 13TN 98FN 15 | Readers (with < 2 years’ experience) without AI.Sensitivity 0.88 (0.82–0.94); specificity 0.88 (0.89–0.96); PPV NR; NPV NRTP 117FP 7TN 104FN 9 | Readers with > 2 years’ experience plus AI. Sensitivity 0.93 (0.91–0.96); specificity 0.92 (0.89, 0.95); PPV 0.92 (0.89–0.95); NPV 0.94 (0.91–0.96)TP 145FP 12TN 144FN 11 | Readers with > 2 years’ experience without AI. Sensitivity 0.88 (0.85–0.91); specificity 0.91 (0.86–0.95); PPV 0.91 (0.87–0.95); NPV 0.88 (0.86–0.91)TP 137FP 14TN 142FN 19 | . | . |
| Salter-Harris | ||||||||
| Nguyen20 | 21/NR | BoneView | Average across 5 radiology residents and 3 expert paediatric radiologistsSensitivity 92.26 (71.29–99.4); specificity NR; | Average across 5 radiology residents and 3 expert paediatric radiologists.Sensitivity 80.95 (57.41–94.78); specificity NR | ||||
Data for hand/wrist fractures were available for BoneView, Rayvolve and Smarturgences (see Table 9). For BoneView, the studies with the most robust sensitivity and specificity data were likely to be Duron17 and Nguyen20 2022 (albeit limited by their retrospective designs). For hand fractures only, and a mixed group of readers, Duron reported sensitivity to be higher with BoneView assistance than without (80.2%, SD 11.4 and 59.6%, SD 20.5 respectively). Specificity was also higher with BoneView assistance than without (91.0%, SD 6.4 and 84.7%, SD 11.0 respectively). Similarly, in Nguyen, for a mixed group of readers interpreting hand/wrist X-rays, sensitivity was higher with BoneView assistance than without (87.08, 95% CI 69.79 to 96.46 and 68.75, 95% CI 49.31 to, 84.32 respectively), although specificity was similar with and without BoneView assistance (88.33, 95% CI 71.34 to 97.1 and 87.92, 95% CI 85.44 to 89.48 respectively). PPV and NPV data were not available for these studies but are provided by Cohen14 in Table 9. For Rayvolve and Smarturgences one the head-to-head study provided subgroup data for hand/wrist fractures. The EAG again highlight the need to interpret these results with caution; the study was a retrospective which included the results of the AI in the reference standard.
For foot/ankle fractures, data were also available for BoneView, Rayvolve and Smarturgences (see Table 9). Again, for Rayvolve and Smarturgences only the limited head-to-head study provided these subgroup data. For BoneView, the studies with the most robust sensitivity and specificity data were again likely to be Duron17 and Nguyen20 (albeit limited by their retrospective designs). For foot fractures only, and a mixed group of readers, Duron reported sensitivity to be higher with BoneView assistance than without (86.9%, SD 8.3 and 71.8% SD 13.6 respectively). Specificity was also higher with BoneView assistance than without (92.9% SD 5.8 and 88.0% SD 9.9 respectively). Similarly, in Nguyen, for a mixed group of readers interpreting foot/ankle X-rays, sensitivity was higher with BoneView assistance than without (70.83, 95% CI 51.47 to 85.89 and 70.83, 95% CI 51.47 to 85.89 respectively). Specificity with BoneView assistance was 86.25, 95% CI 68.77 to 96.02 and without assistance was 85.83, 95% CI 68.26 to 95.79.
Although hip fracture data were available for BoneView and RBfracture (see Table 9), the EAG noted that none of the pivotal studies reported these data. For BoneView, only Oppenheimer21 provided subgroup data for hip fractures, and despite being a prospective study was limited by the inclusion of the AI results in the reference standard. Hip fracture data for RBfracture was either methodologically limited and unpublished (Bonde et al., Radiobotics, 3 July 2024)12 or limited data from a published abstract24 (see Table 9).
Only one study20 provided data for Salter-Harris II fractures in children. As previously noted, although this study may provide reasonable sensitivity and specificity estimates, there is uncertainty surrounding this due to the retrospective study design. BoneView assisted sensitivity, for a mixed group of readers, was given as 92.26, 95% CI 71.29 to 99.4 and unassisted as 80.95, 95% CI 57.41 to 94.78. Specificity was NR (see Table 9).
X-ray reading time
X-ray reading time with and without AI assistance was available for three technologies: BoneView (four studies15,17–19), RBfracture (two studies Yogendra et al., Radiobotics, 3 July 2024,23 though only one study (Yogendra et al., Radiobotics, 3 July 2024) reported data both with and without AI) and Rayvolve (one study22). The data for all fracture types (i.e. not for specific fracture locations) are shown in Table 10. Consistent with the diagnostic accuracy data, the EAG reported these data separately according to the perceived level of experience of readers, as described in the publications. There were no noticeable differences in reading time across the staff groupings, which was surprising given that reading time would be expected to be shorter for more senior staff.
| Less experienced staff | Mixed or unclear staff | Senior | ||||
|---|---|---|---|---|---|---|
| With AI mean seconds per X-ray (SD) | Without AI mean seconds per X-ray (SD) | With AI mean seconds per X-ray (SD) | Without AI mean seconds per X-ray (SD) | With AI mean seconds per X-ray (SD) | Without AI mean seconds per X-ray (SD) | |
| BoneView (Gleamer) | 52.8 (17.7) (Duron)27 (18); range 6–114(Canoni-Meynet)44 (43)(Guermazi) | 61.5 (24.8) (Duron)39 (34); range 4–313(Canoni-Meynet)57 (54)(Guermazi) | 57.0 (49.4) (Duron)49.2 (28.5)([Guermazi)29.6 (19.8)(Meetschen) | 67.0 (59.3) (Duron)55.5 (32.6)(Guermazi)32.2 (20.8)(Meetschen) | 40 (24); range 7–163(Canoni-Meynet) | 50 (28); range 10–180 (Canoni-Meynet) |
| RBfracture (Radiobotics) | – | – | 18.03 (NR) (Yogendra)46.4 (NR) (Bachmann) | 13.6 (NR) (Yogendra) | 11.7 (NR) (Yogendra) | 18.0 (NR) (Yogendra) |
| Rayvolve (AZmed) | 18 (NR)(Fu) | 25 (NR)(Fu) | 19 (NR)(Fu) | 26.1 (NR)(Fu) | – | – |
Studies reported that BoneView and Rayvolve were both associated with a reduction in X-ray reading time across all staff groups: a reduction of 7 seconds per X-ray between means with Rayvolve, while reductions between means ranged from 2.6 to 13 seconds per X-ray with BoneView. One study reported that RBfracture was associated with increased reading time, at 4.4 additional seconds per X-ray between means. However, there were large SDs around reading time in all studies, which the EAG assumed may be due in part to the reading time varying widely according to the type and complexity of the fracture/injury. The EAG was also concerned about the reliability of how reading time would be measured in studies, and potential differences in the way this was defined and recorded between studies. This meant that the EAG had serious concerns about the reliability of the data and how to interpret differences between studies.
The EAG considered that reduced reading time for X-rays may have benefits for service resource, though were uncertain to what extent a difference of seconds per X-ray would mean to a service. However, the EAG also considered it plausible that increased reading times may be acceptable (and preferable) where the additional time translated to increased accuracy. Overall, based on the evidence available, the EAG tentatively concluded that BoneView may be associated with faster reading X-ray reading time, and that these findings should be considered alongside the results for its diagnostic accuracy.
Evidence synthesis
The EAG investigated whether it was possible to meta-analyse data from the included studies, for example, to identify a pooled estimate of sensitivity and specificity for a particular technology. Having identified significant variation in the diagnostic accuracy results according to reader experience, an assessment of the feasibility of meta-analysis was conducted with study results grouped according to the level of experience of readers outlined in Diagnostic accuracy, as well as by fracture and technology type. A threshold of six studies within each grouping was considered sufficient for meta-analysis. The assessment identified that there were sufficient studies for meta-analysis in three categories:
- accuracy of unassisted ED and less experienced clinicians in reading X-rays of any fracture type
- accuracy of unassisted mixed and unclear groups of clinicians in reading X-rays of any fracture type, and
- accuracy of BoneView when assisting a mixed or unclear group of clinicians reading X-rays of any fracture type.
On further investigation, however, there was unexplained heterogeneity in the results of the studies within each analysis, and meta-analysis was therefore not considered to be feasible. Specifically, plots of accuracy data varied significantly and there was a clear positive correlation between logit sensitivity and specificity, which suggested that at least one meaningful covariate was not included in the analysis.30 Within the EVA, it was not feasible to investigate further the potential reasons for heterogeneity in the data, which might have included a meta-regression to investigate factors that influence the accuracy of assisted and unassisted diagnosis.
The EAG took two further steps to synthesise the evidence base: (1) data from the included studies within each grouping was summarised using median and ranges, to give a concise insight into the variability of results across studies and (2) conducted a narrative synthesis to identify patterns in the data that could be used to inform an understanding about the potential value of the technology for assisting in the diagnosis of fractures. Synthesised data from the included studies are split by fracture type (general/all fracture types and specific fracture locations). The EAG advises that the synthesised results provide the median and range of sensitivity and specificity values reported in the studies and do not include any variance (e.g. 95% CIs) around the data reported in the studies. This approach was used for simplicity and, due to limitations in the evidence base as a whole, including unexplained heterogeneity across studies, the results are presented to provide an insight into potential patterns across the data set, rather than to identify precise diagnostic accuracy data for the technologies.
Diagnostic accuracy of the technologies across studies
Median sensitivity and specificity for each of the technologies as reported across study groupings are presented in this section. For ease of interpretation, the results are also presented as the median proportion of missed fractures and FPs. Mixed and subgroup fractures includes the full study population included in the studies, representing multiple fracture types within the case mix of each study, as well as results for specific types of fracture subgroups (hand/wrist, foot/ankle, hip, and non-obvious fractures). Subgroup results (paediatric participants) includes results specific to children and young people.
Mixed and subgroup fractures
Results for unassisted readers are shown in Table 11. The EAG noted that the rate of missed fractures for clinicians reading X-rays without AI assistance was high across studies, even for the senior and expert reader grouping (which was intended to be consistent with consultant-level radiologists and reporting radiographers). Across studies, the proportion of fractures missed ranged from 20% to 30% and was consistent across reader experience, though more experienced readers gave very few FP decisions. The EAG consulted with two radiographers (authors Niamh Gale and Robert Meertens) to enquire whether the accuracy of unassisted diagnosis reported in the papers would be consistent with their expectations in clinical practice. They advised that the rate of missed fractures reported for more experienced staff in the included studies was higher than they expected; they expected that consultant radiologists and radiographers would be expected to miss very few fractures using X-rays (10% or less), even where readers were unable to consult medical notes and the results of other imaging modalities, as was typical in the studies. This would also be consistent with guidelines from The Royal College of Radiologists.31 The EAG sought robust data for the accuracy of X-ray for identifying fractures as used by readers of varying experience but was unable to identify this during the assessment. The EAG considered this to be a significant uncertainty in the evidence base, since uncertainty surrounding the generalisability of the data from the unassisted arms of the studies would also affect the way in which the results for the technology reported in the studies should be interpreted.
| Group (n studies) | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|
| All fracture analyses | ||||
| Unassisted, any staff grouping, all fractures (11) (Yogendra et al., Radiobotics, 3 July 2024)15,17–23,25,26 | 0.72 (0.31–0.93) | 0.89 (0.60–1.00) | 28.1% (7.1–42.1) | 13.3% (1.4–40.0) |
| Unassisted, less experienced staff, all fractures (7)15,17,18,22,23,25,26 | 0.70 (0.58–0.92) | 0.87 (0.60–0.97) | 30.4% (7.9–42.1) | 12.9% (3.0–40.0) |
| Unassisted, mixed or unclear staff, all fractures (9) Yogendra et al., Radiobotics, 3 July 202417–23,25 | 0.73 (0.58–0.87) | 0.90 (0.77–0.97) | 26.7% (13.5–42.0) | 10.7% (2.9–23.0) |
| Unassisted, senior and expert staff, all fractures (3) Yogendra et al., Radiobotics, 3 July 202415,26 | 0.70 (0.65–0.93) | 0.98 (0.96–0.99) | 29.8% (7.1–35.2) | 1.9% (1.4–3.5) |
| Subgroup fracture types | ||||
| Unassisted, mixed or unclear staff, hand/wrist (3)17,20,21 | 0.69 (0.60–0.78) | 0.69 (0.60–0.78) | 35.5% (30.0–40.9) | 14.5% (13.3–15.6) |
| Unassisted senior and expert staff, hand/wrist (2)14,15 | 0.72 (0.68–0.76) | 0.96 (N/A) | 23.5% (N/A) | 3.6% (N/A) |
| Unassisted mixed or unclear staff, foot/ankle (3)17,20,21 | 0.72 (0.54–0.83) | 0.88 (0.86–0.88) | 37.0% (27.3–46.7) | 12.9% (12.5–13.3) |
| Unassisted senior and expert staff, foot/ankle (1)15 | 0.57 (N/A) | NR | NR | NR |
| Unassisted less experienced staff, hip (1)12 | 0.88 (N/A) | 0.88 (N/A) | 11.9% (N/A) | 11.7% (N/A) |
| Unassisted mixed or unclear staff, hip (3)12,21,24 | 0.93 (0.88–0.93) | 0.91 (0.87–1.0) | 9.4% (6.5–12.2) | 11.0% (9.0–13.1) |
| Unassisted less experienced staff, non-obvious fractures (1)16 | 0.31 (N/A) | 0.89 (N/A) | 68.8% (N/A) | 10.5% (N/A) |
| Unassisted mixed or unclear staff, non-obvious fractures (1)21 | 0.72 (N/A) | NR | 28.5% (N/A) | 100.0% (N/A) |
| Unassisted senior or expert staff, non-obvious fractures (1)16 | 0.56 (N/A) | 0.89 (N/A) | 43.8% (N/A) | 10.5% (N/A) |
Sensitivity and specificity each varied significantly across studies though, in general, unassisted readers had higher specificity when identifying fractures than sensitivity. This resulted in a high median rate of missed fractures in all fracture analyses, though the rate of FPs was also over 10% in most studies in all reader groupings except senior and expert readers.
The accuracy of unassisted readers for detecting hip fractures was high. Sensitivity for detecting hand/wrist and foot/ankle fractures was lower than the mixed fracture analyses, and there was variability in specificity for detecting hand/wrist fractures across studies. As might be anticipated, there was greater variability in accuracy across studies reporting findings for groups of readers with mixed levels of experience, which likely represents the variable case mix in readers between studies. There was poorer sensitivity for identifying non-obvious fractures across all reader groupings.
The results for BoneView as evaluated across studies are shown in Table 12. BoneView showed high sensitivity and specificity, irrespective of the reader group. Nevertheless, median numbers of missed fractures (all fracture analyses) exceeded 15% for all readers except the senior and expert reader group. As with unassisted results, there was some variability in sensitivity and specificity values reported across studies, particularly in analyses with mixed and unclear readings. In general, BoneView had improved specificity relative to sensitivity, with fewer FPs than missed fractures.
| Group (n studies) | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|
| All fracture analyses | ||||
| BoneView, any staff grouping, all fractures (7)13,15,17–21 | 0.83 (0.75–0.91) | 0.93 (0.65–0.99) | 17.3% (8.6–25.0) | 7.4% (0.6–35.0) |
| BoneView, less experienced staff, all fractures (3)15,17,18 | 0.79 (0.74–0.89) | 0.97 (0.93–0.98) | 21.3% (11.2–25.7) | 3.3% (1.7–7.4) |
| BoneView, mixed or unclear staff, all fractures (6)13,17–21 | 0.81 (0.75–0.91) | 0.90 (0.65–0.97) | 19.0%(8.6–25.0) | 9.8% (2.6–35.0) |
| BoneView, senior or expert staff, all fractures (1)15 | 0.88 (N/A) | 0.99 (N/A) | 11.7% (N/A) | 0.64% (N/A) |
| Subgroup fracture types | ||||
| BoneView, mixed or unclear staff, hand/wrist (4)13,17,20,21 | 0.89 (0.8–0.96) | 0.92 (0.88–0.95) | 13.3% (8.3–20.5) | 9.4% (7.8–13.3) |
| BoneView senior or expert staff, hand/wrist (2)14,15 | 0.89 (0.88–0.90) | 0.92 (N/A) | 11.7% (N/A) | 7.4% (N/A) |
| BoneView, mixed or unclear staff, foot/ankle (4)13,17,20,21 | 0.89 (0.71–0.98) | 0.93 (0.80–0.96) | 11.6% (1.8–30.0) | 10.4% (6.8–20.8) |
| BoneView, senior or expert staff, foot/ankle (1)15 | 0.83 (N/A) | NR | NR | NR |
| BoneView, mixed or unclear staff, hip (1)21 | 0.93 (N/A) | 0.99 (N/A) | NR | NR |
| BoneView, less experienced staff, non-obvious fractures (1)16 | 0.56 (N/A) | 0.79 (N/A) | 43.8% (N/A) | 21.1% (N/A) |
| BoneView, mixed or unclear staff, non-obvious fractures (1)21 | 0.83 (N/A) | NR | 16.7% | 100% |
| BoneView, senior or expert staff, non-obvious fractures (1)16 | 0.81 (N/A) | 0.89 (N/A) | 18.8% (N/A) | 10.5% (N/A) |
Sensitivity for non-obvious fractures was improved compared to the results for unassisted, although the rate of missed fractures and FPs was still high (43.8% and 21.1%), respectively, suggesting that services may still wish to use precautionary policies to avoid the risk of missed fractures. There was an improvement in sensitivity and specificity for detecting hand/wrist and foot/ankle fractures relative to unassisted. There was very little evidence for the accuracy of BoneView for identifying hip fractures, which reported high accuracy (both sensitivity and specificity), though not conclusively different from some of the unassisted evidence.
Results for RBfracture as reported across studies are shown in Table 13. The sensitivity of RBfracture was broadly comparable to that of BoneView, although there was a trend for specificity to be worse, with similar rates of FPs across reader groupings as were reported for unassisted readers. This may reflect a prioritisation of the technology threshold towards sensitivity; that is prioritising the avoidance of missed fractures over avoiding FPs. However, rates of missed fractures were still generally high. There were fewer studies that evaluated RBfracture, and there was substantial variability in outcomes between studies. This creates more uncertainty in the findings. Notably, one study in senior readers had poor sensitivity compared to other reader groups; although this study was conducted in a paediatric population, which may have influenced the findings [see Subgroup results (paediatric participants)].
| Group (n studies) | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|
| All fracture analyses | ||||
| RBfracture, any staff group, all fractures (3) Yogendra et al., Radiobotics, 3 July 202423,25 | 0.83 (0.70–0.90) | 0.88 (0.67–0.99) | 17.1% (9.7–29.9) | 12.1% (1.1–32.9) |
| RBfracture, less experienced staff, all (2)23,25 | 0.83 (0.70–0.88) | 0.71 (0.67–0.90) | 17.1% (12.2–29.9) | 29.4% (10.0–32.9) |
| RBfracture, mixed, all (3) Yogendra et al., Radiobotics, 3 July 202423,25 | 0.87 (0.80–0.90) | 0.89 (0.85–0.99) | 13.6% (9.7–20.1) | 11.1% (1.1–14.7) |
| RBfracture, senior, all (1) Yogendra et al., Radiobotics, 3 July 2024 | 0.77 (N/A) | 0.88 (N/A) | 22.8% (N/A) | 12.1% (N/A) |
| Subgroup fracture types | ||||
| RBfracture, junior, hip (1)12 | 0.93 (N/A) | 0.94 (N/A) | 7.1% (N/A) | 6.3% (N/A) |
| RBfracture, mixed, hip (2)12,24 | 0.95 (0.93–0.96) | 0.89 (0.86–0.92) | 5.4% (3.7–7.1) | 10.9% (7.7–14.0) |
Results for Rayvolve are shown in Table 14. Two studies evaluated Rayvolve, both of which reported high sensitivity but poor specificity, particularly for hand/wrist and foot/ankle fractures. The EAG considered this was a feature of the technology algorithm, to prioritise missed fractures over FPs. Accordingly, specificity was comparable with unassisted diagnosis, while sensitivity was generally improved.
| Group (n studies) | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|
| Rayvolve all staff all fractures (2)13,22 | 0.94 (0.93–0.96) | 0.83 (0.70–0.85) | 6.1% (4.4–7.4) | 16.9% (14.7–29.6) |
| Rayvolve junior all fractures (1)22 | 0.94 (N/A) | 0.85 (N/A) | 6.1% (N/A) | 14.7% (N/A) |
| Rayvolve mixed all fractures (2)13,22 | 0.94 (0.93–0.96) | 0.77 (0.70–0.83) | 5.9% (4.4–7.4) | 23.3% (16.9–29.6) |
| Rayvolve mixed hand (1)13 | 0.98 (N/A) | 0.75 (N/A) | 2.1% (N/A) | 25.4% |
| Rayvolve mixed foot (1)13 | 0.91 (0.91–0.92) | 0.67 (0.63–0.72) | 8.0% (7.1–8.9) | 32.8% (27.9–37.7) |
Results for TechCare Alert are shown in Table 15. Two studies evaluated TechCare Alert, with no crossover in the reader groupings, meaning that data were only available separately or in a group reporting results for any reader grouping. Both studies reported high sensitivity and specificity for TechCare Alert, with no rates of missed fractures and FPs. Results were improved compared to unassisted diagnosis. Results were similarly high for hand/wrist and foot/ankle fractures.
| Group (n studies) | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|
| TechCare Alert all staff all fractures (2)13,26,a | 0.93 (0.90–0.95) | 0.98 (0.93–0.98) | 7.1% (5.1–9.8) | 1.9% (1.9–7.5) |
| TechCare Alert junior all fractures (1)26 | 0.95 (N/A) | 0.98 (N/A) | 5.1% (N/A) | 1.9% (N/A) |
| TechCare Alert mixed all fractures (1)13 | 0.90 (N/A) | 0.93 (N/A) | 5.1% (N/A) | 1.9% (N/A) |
| TechCare Alert senior all fractures (1)26 | 0.93 (N/A) | 0.98 (N/A) | 7.1% | 1.9% |
| TechCare Alert mixed hand (1)13 | 0.94 (N/A) | 0.92 (N/A) | 6.2% (N/A) | 8.3% (N/A) |
| TechCare Alert mixed foot (1)13^ | 0.88 (0.85–0.90) | 0.91 (0.90–0.92) | 11.9% (9.5–14.3) | 8.6% (7.9–9.2) |
Subgroup results (paediatric participants)
Median results across studies reporting diagnostic accuracy data in children and young people only are shown in Table 16. No evidence was available in a less experienced reader group only. Among mixed or unclear experience readers, the assistance of BoneView or RBfracture improved median sensitivity for detecting fractures, though made no clear difference to specificity. In real terms, this reduced the number of fractures that were missed but did not change the number of FPs. One study that reported data for highly experienced staff reported poor sensitivity both with and without assistance with RBfracture, which lacked face validity. In this group of readers, the assistance of RBfracture improved sensitivity, but this came with a cost to specificity; that is there was a meaningful reduction in missed fractures but a large increase in FP diagnoses.
| Group (n studies) | Technology | Staff | Fracture | Median sensitivity (range) | Median specificity (range) | Median % missed #s (range) | Median % over diagnosis (range) |
|---|---|---|---|---|---|---|---|
| No AI mixed all fractures (4) Yogendra et al., Radiobotics, 3 July 202420,21,23 | No AI | Mixed | All | 0.78 (0.73–1.0) | 0.91 (0.77–0.95) | 22.6% (0.0–26.7) | 9.3% (5.6–22.5) |
| BoneView mixed all fractures (2)20,21 | BoneView | Mixed | All | 0.91 (0.83–1.0) | 0.91 (0.30–0.92) | 8.7% (0–17.3) | 9.0% (8.0–10.0) |
| RBfracture mixed all fractures (2) Yogendra et al., Radiobotics, 3 July 202423 | RBfracture | Mixed | All | 0.90 (0.89–0.90) | 0.90 (0.80–0.99) | 9.9% (9.7–10.2) | 10.8 (1.1–20.41) |
| No AI senior all fractures (1) Yogendra et al., Radiobotics, 3 July 2024 | No AI | Senior | All | 0.65 (N/A) | 0.99 (N/A) | 34.2% (N/A) | 1.4% (N/A) |
| RBfracture senior all fractures (1) Yogendra et al., Radiobotics, 3 July 2024 | RBfracture | Senior | All | 0.77 (N/A) | 0.88 (N/A) | 22.8% (N/A) | 12.1% (N/A) |
Conclusions of the clinical, service and technological evidence
As an emerging technology, the evidence base for the clinical and service value of the technology for assisting with fracture diagnosis was expectedly limited. Nevertheless, within the context of an EVA, the EAG considered it notable that a total of 16 studies were identified and eligible for inclusion in the evidence review. Within the methods of the EVA, the EAG was appropriately broad in its inclusion criteria: including conference abstracts, non-peer reviewed reports, and manuscripts in preparation, all of which may not typically be considered within a NICE evaluation. The EAG also included a variety of study designs, including those with known limitations for determining reliable estimates for diagnostic accuracy. As the aims of an EVA include identifying an overview of the existing evidence base and key evidence gaps and research needs, the inclusion of this evidence was useful for these purposes. However, as stressed within the report, these studies are not robust for identifying reliable estimates of clinical and service outcomes and should be interpreted with caution.
The evidence base as a whole suggested that there is a need for further, high-quality research to evaluate the clinical and service outcomes associated with the technology (see Chapter 9). While several of the technologies had been evaluated in few studies available to date, there were nine and five included studies evaluating BoneView and RBfracture. For these technologies, the EAG considered that the time for a proof-of-concept of the technologies had passed, and that these technologies require evaluation within robust comparative study designs. These would include diagnostic randomised controlled trials (RCTs) and prospective, robustly sampled diagnostic studies, each set within the likely target settings and reader groups that would be expected to use the technology in clinical practice. To date, much of the evidence has focused on the accuracy of the technology as a stand-alone tool (excluded from the evidence review as clinical standards would mean that this use would not be possible within the NHS) and as assistance to clinicians in radiology departments, including consultant-level staff, who would not be expected to require the use of the technology in clinical practice. In addition to the potential value of the technology for avoiding missed fractures, stakeholders to the assessment highlighted the potential value of the technology for service outcomes, such as time and resource savings, but this has not been a focus of any of the available studies to date. All this said, however, the EAG noted that the evaluation of the technology will be particularly complex in this field, as compared to evaluating diagnostic tools in other indications (see Chapter 9). In the same way as these complexities affected the ease of interpretation of the evidence base in this assessment, the EAG expected that some of these complexities would remain for any future NICE assessment, even with a more developed evidence base.
Given the limitations in the included studies, the EAG considered that the precise estimates of sensitivity and specificity reported in the studies were uncertain and may have limited generalisability to the outcomes that may be seen in clinical practice. However, across the evidence identified, there was a trend for the technologies to result in improvements in diagnostic sensitivity for identifying fractures relative to unassisted diagnosis, but minimal or no improvement in specificity for identifying where fractures were not present. This trend was seen across technologies and reader groups. Where accuracy of diagnosis unassisted by the technology was already high, such as in the identification of hip fractures and when used by senior readers with high accuracy, the additional value of the technology was evidently smaller. Generally speaking, the evidence showed that the use of the technology did not result in perfect sensitivity and specificity for any reader or any fracture type: while not unexpected, the implications of this were that the technology could not be relied upon to identify all fractures or to never result in a FP diagnosis. Within analyses of non-obvious fractures, where arguably the use of the technology could be greatest, the technology generally improved sensitivity but was still associated with a large number of FNs that would require handling through additional service use (e.g. imaging, consultant review, precautionary tactics, patient recall). While studies mostly reported that use of the technology may reduce reading time for each X-ray by several seconds, advice from clinical experts was that this may not be preferable, as the technology should – using best practice – be an add-on step of the care pathway and thus result in overall increased time for diagnosis within services.
In conclusion, on the basis of the available evidence base, the EAG considered that there are early indications that the use of the technology to aid identification of fractures could have value for avoiding missed fractures. Further evidence is needed to evaluate whether this potential holds true for the specific ways in which the technology could be used within the NHS, including the target fracture types, readers, and the broader care pathway used within each of the target settings. There was some evidence that accuracy may vary across the technologies evaluated; however, due to the lack of evidence for some technologies, and variation in study designs used to evaluate each technology, this was uncertain. There is also a need to determine the potential implications of the technology for broader service use outcomes, particularly to determine any potential trade-off between increased time to diagnosis with other potential time and resource savings. Further evidence is also needed to determine the clinical outcomes associated with any difference in diagnostic accuracy and service use.