The EAG conducted a broad evidence review to identify the existing evidence base for the use of the technology to assist with the identification of fractures in emergency care settings. The EAG identified:
- 16 studies that evaluated the diagnostic accuracy of the technology
- 7 studies that reported X-ray reading time
- 0 studies that reported other service outcomes
- 0 studies that reported clinical outcomes for people with suspected fractures
- 0 economic evaluations of the technology.
With respect to the specific technologies eligible for consideration, the majority of the evidence evaluated BoneView (nine studies) and RBfracture (five studies), with two studies available for both Rayvolve and TechCare Alert, and 0 studies available for qMSK. The availability of a head-to-head comparison13 of three of the technologies this early in development of the technologies was notable, although this study lacked the availability of an unassisted arm for comparison purposes. As discussed throughout the report, the evidence base to date was limited in quality, with various risks of bias to the results as well as concerns about the generalisability of the evidence to clinical settings. Due to these issues, the EAG was unable to draw firm conclusions about the potential value of the technology, or identify reliable patterns in the results, such as according to reader experience, case mix, fracture subgroup, or population.
During the assessment, including during interpretation of the evidence base and feedback from clinicians and stakeholders to the assessment, the EAG identified complexities in developing evidence to evaluate interventions for use in people with suspected fractures and the services that treat them. One of the most significant of these considerations was that the target population of people with suspected fractures is highly heterogeneous, comprising of people with a broad range of demographics, type, mechanism and location of injury, and broader health considerations. As the diagnostic accuracy of X-ray and the broader care pathway will vary across populations (including the diagnosing clinicians, use of additional imaging modalities, use of precautionary tactics, and ongoing treatments), this means that the potential value of the technology will vary according to the population in which its used. The evaluation of the technology within a representative population will therefore be key to deriving reliable estimates for outcomes. However, the target population is also not static, and will vary between types of urgent care settings (ED, UTC, MIU), across geographical areas in the UK, and will vary even within the same centre according to the day of the week, time of day, or season. Each centre will also vary in the local policies that they use to diagnose certain types of fractures and fractures in certain subpopulations, including their use of precautionary tactics but also including the typical staff available to read X-rays and the length of time until a definitive diagnosis is made. To inform understanding of the potential value of the technology, it will therefore be important to understand the way in which outcomes for the technology change according to differences in the population case mix, reader, and care pathway.
The economic analysis contained within this report represented a very top-level overview of the likely costs and consequences of adopting AI-assisted diagnosis of fracture within urgent care settings. Due to the resource and time constraints of this EVA, the EAG was unable to explore a large number of issues and nuances apparent in the data and a substantive evidence synthesis project of 12–18 months’ duration would, in the opinion of the EAG, provide a solid appraisal of the evidence to fully inform a decision as to whether AI-assisted diagnosis of fracture represented a value-for-money investment in the NHS. Particular issues this should consider include:
- Evaluation of the diagnostic accuracy, clinical and service outcomes associated with the technology within robust study designs within settings comparable with the likely use of the technology in clinical practice.
This should include diagnostic RCTs and prospective, robustly sampled comparative studies (assisted vs. unassisted diagnosis) published in peer-reviewed formats. Given that the technology would be expected to influence both clinical and service outcomes, future studies are needed to evaluate outcomes across both of these domains.
- Formal network meta-analysis of studies and/or head-to-head studies.
A formal network meta-analysis was not considered feasible due to heterogeneity between studies (see Evidence synthesis). However, further work is required to determine whether a less formal synthesis could be conducted, or else head-to-head studies are required for all relevant AI algorithms/software.
- Studies to evaluate the factors that influence the value of the technology for identifying fractures.
This may include studies designed to explore change in outcomes according to key factors that would inform use of the technology, such as reader experience, case mix, the way in which AI is integrated into the patient pathway, and determinants of patient outcomes, such as patient age, frailty, and prevalence of health conditions affecting bone health. When a suitable evidence base is available, meta-regression to explore factors that influence outcomes may help decision-makers to target the use of the technology in clinical practice.
- Analysis of optimal cut-off points.
This analysis made use of the stated sensitivity and specificity estimates from the literature. However, manufacturers of the algorithms (as well as the NHS) should consider the most cost-effective cut off score to maximise the efficiency of diagnosis. For example, some algorithms will generate a propensity score or probability of an X-ray being a fracture. An internal setting will determine what score and above is defined as a positive. Varying this score varies the sensitivity and specificity (from which the receiver operating characteristic curve can be generated). Connecting this to a decision model allows estimation of the optimal cut-off score for an algorithm (Laking et al.49)
- Greater exploration of the likely longer-term costs and consequences of true and false-positive and -negative diagnoses.
This analysis relied on published studies to approximate long-term costs and consequences of the four outcomes from a diagnosis. These varied in comparability, for example, in terms of scope1 of resource use included and time horizon. A more comprehensive model breaking these items down in greater detail would enhance comparability between the different fracture types.
- Impact of detecting multiple fractures.
The economic analysis did not differentiate between a single and multiple fracture in a single patient. A particular use case and benefit of AI-assisted diagnosis could be in identifying a less obvious additional fracture which may be more likely to be missed by the reader. Further research into the benefit of this is warranted.
- Second read of only positives or only negatives.
Scenario 9 of the economic analysis only considered a second read/review of all X-rays. An alternative approach is to review all positive or all negative diagnoses alone. Additional modelling would facilitate exploring the cost-effectiveness of this which may assist in enhancing the efficiency of the diagnostic X-ray service in clinical practice.
- Formal assessment of study quality.
Consistent with methods for an EVA, formal quality assessment of the included studies was not conducted, and quality limitations of the included studies was conducted informally and discussed throughout the report. However, formal quality assessment of the current and future evidence base would be useful for characterising the strength of the evidence and identifying key weaknesses and their effect on outcomes. For the assessment of diagnostic accuracy studies, QUADAS-AI,50 an extension to the original tool to account for the considerations specific to AI technologies, would be useful.