The decision problem for this assessment is described in the National Institute for Health and Care Excellence (NICE) scope1 and External Assessment Group (EAG) comments and planned assessment methods are included in the protocol.2
During its assessment, the EAG made the following minor adjustments to the planned methods outlined in the protocol.2 These were:
- Definition of frailty: none of the included clinical effectiveness evidence reported the proportion of participants who were assessed as being frail, had experienced a frailty fracture, or reported outcomes specific to this group. Though one of the included economic studies (Beaupre et al.3) stated that frailty was considered and assessed bone mineral density for elderly people when needed, it did not define ‘frailty’ and indicated that only age was used to classify fragility fractures.
- To help characterise the prevalence of people with frailty who were included in the evidence, the EAG reported other metrics that were imperfect but approximate indicators of frailty in the sample, where reported. This included the proportion of participants aged ≥ 80 years and the proportion of injuries due to falls. These indicators were considered to be imperfect, however.
- Reference standard: the EAG included evidence from studies that used a reference standard that did not match that described in the NICE scope1 and review protocol2 (i.e. definitive report from a consultant radiologist or reporting radiographer). These decisions were taken due to a paucity of high-quality evidence directly relevant to the decision problem and due to uncertainty surrounding the correspondence between staff grades in other healthcare systems compared to consultant radiologists and reporting radiographers in the NHS. Where reference standards were considered to be indirect or flawed in some way, this is highlighted in the report.
- Screening and prioritisation of evidence: the review protocol2 specified that the EAG would prioritise the clinical and economic evidence and outcomes that best addressed the decision problem for this assessment. As there were numerous quality considerations across the available evidence, it was not possible for the EAG to select a group of robust studies for priority inclusion in the review. Accordingly, the EAG included all of the evidence identified that reported diagnostic accuracy data for the included technologies, despite its limitations. As discussed later in the report, the limitations with the evidence means that the EAG did not aim to determine the diagnostic accuracy of the technologies, but rather to provide an overview of the existing evidence base, initial interpretations from the results from patterns across the data, and recommendations for future research. One study4 was de-prioritised following identification, as it did not report any of the priority outcomes in the protocol.2 This study assessed healthcare professional user acceptability of artificial intelligence (AI) for the detection of fractures before and following its implementation in a healthcare setting.
- Selection of clinical evidence to inform the economic model: while all studies were associated with key limitations in quality, the EAG sought to select estimates of prevalence, sensitivity and specificity that could be used to inform the economic analysis. The studies with the most robust data – relative to the evidence base as a whole – were selected, though these were nevertheless considered to be unreliable. Studies with the following design features were prioritised for selection of sensitivity and specificity: studies with larger sample sizes; studies reported in peer-reviewed publications; studies that reported results for both AI-assisted and -unassisted clinicians; and studies that used a reference standard that did not include the results of the AI. Prevalence data selected for use in the economic analysis were derived from studies with: robust sample selection processes; larger samples; and relevant eligibility criteria. As this was an early value assessment (EVA), these studies were not subjected to a formal quality appraisal; however, key limitations of included studies are discussed within the report.
- Information sought from companies: the EAG submitted clarification questions to four of the relevant companies (Gleamer, Radiobotics, AZmed, Milvue) on 29 July and companies were asked to return their responses by 7 August. An additional round of questions was submitted to one company (Radiobotics) on 5 August. These dates were later than scheduled in the protocol2 to coincide with the completion of evidence selection, at which point the questions could address uncertainties across the identified evidence base.