Statement of principal findings
Systematic review of RAMs
Sections of this chapter have been reproduced from Pandor et al. under licence CC BY 4.0.17 In a systematic review of 51 observational studies evaluating RAMs for predicting the risk of developing VTE in hospital inpatients, we found that VTE RAMs have generally weak predictive accuracy. The studies validating these models are heterogeneous and most have a high risk of bias. Lack of methodological clarity was common, leading to difficulty in assessing the applicability of the individual study results. VTE prophylaxis was used in around half the study populations, with markedly varying frequency, and was not reported in most of the other studies. This is an important potential source of bias and heterogeneity between studies.
Our systematic review updated previous reviews to include more recent studies, but did not fundamentally alter the conclusions that the available evidence shows weak predictive accuracy and has significant methodological limitations.8–10 VTE RAMs may be useful, despite the limited evidence and apparently weak accuracy, if they can identify patients with a higher risk of VTE who are more likely to benefit from prophylaxis. Decision-analytic modelling allowed us to explore whether VTE RAMs can provide more cost-effective use of VTE prophylaxis.
Economic evaluation
Using decision-analytic modelling to address the cost-effectiveness of VTE risk assessment, we made several key findings. For medical inpatients, the balance of VTE risk and risk of major bleeding, combined with the performance observed for RAMs in medical cohorts, means that prophylaxis for all is the optimal strategy within the model. The key factors determining cost-effectiveness in the medical cohort appear to be the baseline risks of VTE and major bleeding and the increased risk of major bleeding associated with prophylaxis. In the analysis for surgical inpatients, the optimal strategy within the model is to offer prophylaxis to all. However, the scenario analyses found that the optimal strategy was sensitive to many of the individual model inputs and assumptions tested, with the optimal strategy varying from a Pannucci score of ≥ 6 (sensitivity of 40% and specificity of 82%) to offering prophylaxis for all.
In the scenario analyses exploring specific surgical populations who require longer durations of prophylaxis, it was found that prophylaxis for all is optimal in patients having elective knee replacement whereas offering prophylaxis at a Pannucci score of ≥ 3 (sensitivity of 84% and specificity of 49%) was optimal when offering extended duration prophylaxis with LMWH to patients having elective hip replacement. The difference in optimal strategy between the elective hip replacement population and the main surgical cohort appear to be driven in part by the high cost of providing extended-duration prophylaxis, as the scenario analysis assuming 28 days of LMWH in the general surgical population also resulted in a Pannucci score of ≥ 3 being optimal. In the elective knee replacement population, the increased risk of bleeding associated with LMWH was lower meaning that there was less of a trade-off between a reduced VTE risk and an increased bleed risk in this group.
Our findings differ slightly from the previous cost-effectiveness analysis from a US health system perspective by Le et al., which found that LMWH is cost-effective for medical inpatients when the risk of VTE is over 1%.11 Le et al. discussed the use of RAMs to select patients with a risk of < 1% but did not explicitly model using a RAM to identify patients with a risk of < 1%. The difference is explained by the fact that Le et al. only included proximal symptomatic DVT and PE within their analysis and their average VTE risk in medical inpatients was assumed to be 1.4%. Under these conditions, the optimal strategy using our model would be a Padua score of ≥ 3 (when valuing a QALY at £20,000 or £30,000). Green et al. report a risk of < 1% in those with a Padua score < 3, suggesting the analyses are consistent when the VTE risk is defined in a similar manner.39
Assessing the feasibility of efficient data methods and estimating key parameters for future implementation research
The findings from our cohort study suggest that currently available, routine and efficient data methods which can potentially identify VTE or major bleeding events during hospital admission or within 90 days of discharge, are not sufficiently sensitive for use in a large data-enabled study. We failed to achieve feasibility for a number of predefined metrics and concluded that use of routine data to identify outcomes would be highly likely to miss important VTE and bleeding events, and may erroneously identify false-positive events. We also identify significant limitations in the ability of efficient methods to identify individual risk variables and facilitate RAM comparison at scale in any future work. The majority of our sites did not collect contemporaneous data on risk assessment in a digital, or easily accessible format.
In addition, we provide estimates of key parameters for any future implementation study, based on an observational cohort study of > 2000 patients across four NHS sites. We report a HAT event rate of 1.6% (95% CI 1.0% to 2.2%) and a major bleeding event rate of 2.5% (95% CI 1.8% to 3.2%) in the context of routine NHS care for hospital inpatients. We also identify high rates of pharmacological prophylaxis in the context of routine care and risk assessment using the Department of Health tool, noting prescribing rates of 86.2%, 80.4% and 93.7% for all, medical and surgical patients, respectively. When comparing RAMS with attempted validation, we demonstrate highly variable rates of potential pharmacological thromboprophylaxis within the same cohort of general medical and surgical patients, in a UK NHS setting. Prescribing recommendation rates at published thresholds varied from 13% to 91% between RAMs. These data provide key estimates of current and potential prescribing rates, and potential event rates for any future studies looking to compare VTE RAMs, or assess implementation of novel strategies.
Finally, recent survey work suggests that > 80% of UK NHS sites continue to use the Department of Health tool for VTE risk assessment and that a high proportion use weight-adjusted dosing of pharmacological thromboprophylaxis. Our survey data also found that many sites continue to conduct risk assessment via paper proforma only and that very few sites record the absence of risk characteristics. These findings suggest that further comparative research on individualised risk assessment reliant on routine data collection, is unlikely to be feasible or reliable. These are important points to consider in the design of any future study.
Strengths and limitations
Systematic review of RAMs
Our systematic review work has a number of strengths. The review was conducted with robust methodology in accordance with the PRISMA statement and the protocol was registered with the PROSPERO register. Clinical experts were involved throughout as checkers and to assess the validity and applicability of research during the review. We reported descriptive statistics to provide insight into the limited evidence base applicable to the subject matter, and the scientific concerns regarding validity of the data. However, there are a number of potential weaknesses. Decisions on study relevance, information gathering and validity were unblinded and could potentially have been influenced by pre-formed opinions. However, masking is resource intensive with uncertain benefits. The studies of risk prediction were a combination of prospective cohorts and retrospective health database registries. Both have significant limitations. Retrospective studies of health database registries may have large numbers but may be limited by poor data quality and failure to accurately ascertain outcomes. Prospective cohorts may have better quality data but with smaller numbers lack statistical power.
We were unable to undertake meta-analysis or statistical examination of the causes of the observed heterogeneity. Potential sources of heterogeneity include variation in study design, the study population, how RAMs are implemented, outcome definition and measurement and the use of thromboprophylaxis. The latter point is highlighted above as a key finding.
Economic evaluation
The main strength of the economic evaluation is that it is able to evaluate the trade-off between costs, benefits and harms in a systematic manner based on the existing evidence base. One of the key uncertainties in the modelling is the incidence of VTE in cohorts not having thromboprophylaxis and the incidence of major bleeds outside of clinical trials. To address this uncertainty, the two-way scenario analyses on VTE and bleeding risk provide information on how robust the conclusions would be if the average risk of bleeding and the average risk of VTE across the medical and surgical cohorts has been underestimated or overestimated. However, these analyses do not necessarily provide information on whether a different strategy would be optimal in a select cohort of patients known to have higher or lower risks of bleeding or VTE. This is because the performance of the RAMs has been measured in broad cohorts of patients and the RAMs may not perform as well in very select cohorts chosen to have a very high or very low risk of VTE or major bleeding.
The economic analysis assumed that the population did not have characteristics associated with an increased risk of bleeding, and therefore had a low risk of bleeding overall. The findings, and specifically the conclusion that prophylaxis for all is the optimal strategy, do not, therefore, apply to populations or individual patients with an increased bleeding risk. The analyses showing whether the overall health impact is positive or negative at varying levels of VTE and bleeding risk (see Figures 11 and 12) are intended to show how bleeding and VTE risks might interact but should not be used to guide individualised decisions for patients. Bleeding and VTE often share the same risk factors, and interactions may occur between risks and between risk factors. Furthermore, our analyses assume that the RR for bleeding associated with prophylaxis is constant and increases in proportion to baseline bleeding risk. It is conceivable, however, that the risk factors for bleeding could increase the RR associated with bleeding. Taken together, these issues mean that the decision to provide VTE prophylaxis in the presence of increased bleeding risk is extremely complex, and best undertaken on an individual patient basis by an experienced clinical decision-maker. In this scenario, there remains a potential option to use mechanical thromboprophylaxis through graduated compression stockings or intermittent pneumatic compression devices. However, the evidence to support use of either device in medically ill patients is limited. We opted not to evaluate alternative methods of mechanical prophylaxis for patients at high risk of bleeding within the model, for the reasons outlined in Chapter 4.
A key limitation of the economic analysis is the heterogeneity in the estimates of RAM performance across the various cohorts identified in the systematic review. The secondary analysis using performance data for the Padua RAM from the cohort reported by Elias et al. provides different conclusions about the optimal strategy than found in the base-case analysis, due to the better performance of the Padua RAM in this cohort compared to the performance of this and other RAMs in other cohorts. The Padua RAM did not perform consistently better than other RAMs in other studies, so the higher accuracy in the Elias study probably reflects study characteristics rather than superiority of the Padua RAM. Our use of these data was intended to explore the potential for an accurate RAM to be cost-effective, rather than draw conclusions regarding a specific RAM or population.
Estimating key parameters for future implementation research and assessing the feasibility of efficient data methods
We engaged a combination of digitally mature and paper-based NHS sites within our cohort study and supplemented the findings with survey work, to obtain an accurate picture of VTE risk assessment conduct in a contemporary UK setting. We based key parameter estimation on an updated systematic review (workstream 1) and strict consensus definitions for VTE and bleeding events. We used statistical support, topic experts and research staff to iteratively develop our data collection tool and designed a clear workflow diagram to ensure compliance with data collection methods and limit subjective interpretation of case note data.
However, there are a number of potential limitations with design which could impact on the outcomes from workstream 2. We recruited only teaching hospital sites with university affiliation, two of which were VTE exemplar centres and three of which were tertiary centres, which may limit the external validity of our key parameter estimates. Regarding the evaluation of risk variables and estimation of key parameters, we opted for a cohort design on the early assumption that VTE risk assessments would be completed prospectively and accurately at all sites, in line with NICE guidance and NHS contract standards,86 and that these data would be retained within the healthcare record. Limitations in electronic data capture at several chosen sites led to this aspect of the project becoming single centre. As such, we were unable to undertake detailed work to determine whether other sites could implement efficient methods of recording risk assessment data, but the issues identified with limited IT and service pressures suggested that substantial work would have been required and success was unlikely.
In addition, the complexity of identified RAMs for comparison necessitated retrospective case note review to identify additional variables. This may have introduced bias in assessment and downgrading of other RAMs due to absence of inquiry and information in the case notes. Other limitations include a relatively small sample size and differential ascertainment for the Department of Health VTE risk tool (collected in real time by clinicians) compared to other RAMs (collected through case note review by research teams). In addition, we do not report estimation of bleeding risk within this work. Although this is a vital aspect of VTE risk assessment, the six RAMs presented for comparison do not integrate any form of bleeding assessment. It is therefore unclear how these RAMs would perform comparatively within a real-world population of medical and surgical inpatients. It is likely that the proportions of patients receiving prophylaxis in practice would be lower than those recommended once bleeding risk is taken into account. The exclusion criteria applied to this cohort limit the generalisability of our results.
Regarding the feasibility of efficient data methods, research assistants across sites varied in seniority and clinical experience. Although all sites had a principal investigator with clinical expertise available for queries and strict working definitions for outcome events, this may have introduced variation in reporting. Finally, we did not achieve our intended target of 3000 patients. Reasons for this are detailed in the above chapters. However, it is important to note that the overall results within our cohort of 2008 patients are well outside of feasibility targets and that sensitivity values using routine data sources to identify key outcomes are universally poor. We do not envisage that further cases would have significantly affected these values to the point of reaching feasibility, even with a high level of accuracy.
Research in context
Systematic review of RAMs
The present review is the largest and most comprehensive systematic review in this field to date. It includes 18 recent studies published since the completion of the previous systematic review. These studies are consistent with the previous literature in that they report modest performance of the assessed RAMs, with limitations in methodology and reporting preventing further analysis. The conclusion of this review therefore concurs with previous systematic reviews: there is insufficient evidence to recommend one RAM over another.
Economic evaluation
Although the optimal strategy of prophylaxis for all is the same in the medical and surgical cohorts, there is a more evident trade-off between the benefits of reduced VTE and the risks of increased bleeding for surgical patients as shown by the table of clinical outcomes in Table 4. This is why the optimal strategy is more sensitive to changes in the model inputs and assumptions in the surgical population. Parameters related to the long-term benefits of preventing VTE, such as the risk of PTS and the utility decrement of PTS, were identified as being important in the EVPPI analysis and sensitivity analyses for the surgical cohort. Factors related to long-term risks of chronic complications such as PTS do not appear to be important in determining the optimal strategy in the medical cohort, which may be because the medical cohort is older and, therefore, has a shorter life expectancy over which to accrue benefits from avoiding long-term complications. It is also important to recognise that while the risk of any major bleeding is higher in the surgical population, the risk of fatal bleeding or a non-fatal ICH is higher in the medical population. Medical inpatients are also more likely to die following a PE, and more likely to die in the year following admission even without experiencing VTE or major bleeding. However, sensitivity analyses applying a lower case-fatality rate for PE in medical inpatients or a higher case-fatality rate for PE in surgical inpatients do not result in a change in the optimal strategy.
In the secondary analysis using RAM performance data from the study by Elias et al., the optimal strategy was to use a RAM in both medical and surgical patients. This is because RAM performance from Elias et al. was better than assumed in the base case for either the medical or surgical patients. For example, Elias et al. reported a sensitivity of 99.9% and a specificity of 23.7% for a Padua score of ≥ 3, whereas the equivalent values reported by Greene et al. in an exclusively medical cohort, were 49.3% and 73.0%, respectively. Performance data for the Padua RAM were not evaluated in an exclusively surgical cohort so a direct comparison cannot be made. These results suggest that if a RAM were to be useful for determining which patients should receive prophylaxis, it would need to have very high sensitivity similar to that observed for the Padua RAM in the study by Elias et al. If a future RAM performed similar to the Padua score ≥ 3 and a cut-off of 3 was applied, the model predicts that we would expect 80% of surgical patients and 84% of medical patients to receive prophylaxis. NICE reported that according to national figures, over 70% of medical patients receive prophylaxis using the Department of Health Risk Assessment tool, with some trusts offering prophylaxis to over 90% of medical inpatients.8
Estimating key parameters for future implementation research and assessing the feasibility of efficient data methods
Our findings are in line with previous comparative research on VTE risk assessment. Systematic review data have reported prescribing recommendation rates ranging from 30% to 90% for medical patients with the application of different RAMs.10,145,152,153 We have not identified previous work describing the comparative prescribing rates of different RAMS in a cohort of surgical patients and therefore provide new comparisons in this area. The baseline data on hospital length of stay, HAT event rates, major bleeding rates and prescribing rates are all in keeping with published evidence reviews, suggesting internal validity of our cohort.8
The findings in our study highlight the ongoing variation in clinical care, of both medical and surgical patients, when different RAMs are applied to similar populations. In addition, our findings support previous research regarding the risk thresholds employed using the Department of Health VTE risk tool; this RAM conferred the highest proportion of pharmacological prophylaxis to both medical and surgical patients than any of the other six RAMs, with a 36% increase in recommended prescribing for surgical patients, and a 21% increase in medical patients, compared to the next highest prescribing model. Such high prescribing recommendations invite two key questions. First, does this tool provide better clinical or cost-effectiveness compared to others? Second, if over 90% of patients are likely to receive pharmacological thromboprophylaxis following use of the RAM, is there any real value in asking clinical teams to perform subjective and complex risk assessment? Switching to an ‘opt-out’ approach based on contraindications and bleeding risk has the potential to confer advantage, through mitigation of time and opportunity costs while concurrently encouraging default and reliable prescribing options. Such changes could also mitigate the key recurring issues with preventable HAT, most commonly resulting from errors in assessment, linked prescribing or drug administration.150
Our findings of limited sensitivity using routine data methods within a UK setting are consistent with previous international work in this area. An early comparison of HES data to general practice records in England reported in 2012, initially concluded reliable identification of vascular disease using HES data (derived from ICD-10 coding data).154 However, this analysis was restricted to PE from a VTE perspective and sought only to correlate disease states, rather than identify new case episodes. Several authors have also used primary care research data sets correlated to evidence of anticoagulation or other secondary care data to identify VTE events, with reported reliable capture. This work does not seek to discriminate between index presentation of VTE and HAT.155,156 A systematic review, with searches run in July 2010, published in 2012, summarised findings on this topic from 19 studies. The positive predictive value (PPV) for PE ICD codes ranged from 24% to 92%, with higher values from certain combinations of codes. PPV values for DVT codes ranged from 31% to 97%. More recently, a cross-sectional North American study compared ICD-10 codes for VTE in hospitalised medical patients to a ‘gold standard’ manual review of clinical data in 4000 patients.157 The authors report a sensitivity of 63%/PPV of 43% for any DVT and a sensitivity of 83%/PPV of 70% for PE, implying further discrepancy between types of VTE.
Several authors have also experimented with composite data sets and diagnostic/procedural/disease coding combinations. One study combined ICD codes for VTE with a common procedural terminology code for a VTE Diagnostic Study plus at least one of the following within 30 days of diagnosis: pharmacy script for anticoagulation, placement of an inferior vena cava filter, or death.158 This algorithm still lacked sensitivity, reporting a value of 0.67 (0.60, 0.73), a specificity of 0.99 (0.98, 0.99), a PPV of 0.92 (0.87, 0.96) and a NPV of 0.94 (0.93, 0.95). Alotaibi et al. subsequently combined routinely collected ICD-10 coding data with imaging procedure codes to identify VTE events over a 10-year period compared to case note review and reported highly specific results but limited sensitivity, in line with our findings [74.83% (95% CI 67.01 to 81.62) and 75.24% (95% CI 65.86 to 83.14) for PE and DVT, respectively].159 Verma et al. report findings using natural language processing (NLP) algorithms for digital interrogation of radiology reports in a large cohort of hospitalised medical patients.157 The authors conclude a high level of accuracy for NLP algorithms, reporting sensitivities of 94%/91% and PPVs of 90%/89% for DVT and PE, respectively. Such algorithms would require external validation in a UK setting and would be unable to distinguish HAT from index presentation of VTE, in isolation. Reliance on such algorithms in the context of low event rates would also lead to potential bias from false-negative results.
In 2017, Baumgartner et al. highlighted further issues through interrogation of an administrative coding database, looking to determine the accuracy of ICD coding for new episodes of recurrent VTE in patients with a prior history.160 Only 31.1% of coded encounters were verified by reviewers as true recurrent VTE. More recently, Pellathy et al. have conducted similar work to VTEAM within the USA, comparing accuracy of HAT diagnoses made through administrative coding to manual case note and radiology review.161 The authors report only 40% of HAT cases identified through routine coding were confirmed by case note review and 45% of HAT confirmed through diagnostic test records lacked corresponding ICD codes. This body of evidence, now including our recent work, suggests caution with the use of routinely collected outcome data within any future study where HAT events are an outcome of significant importance.
There are multiple potential explanations for the limited performance of routine data to identify HAT. The condition is a temporal phenomenon and routine coding data can therefore mistake index presentation with VTE as HAT, or fail to code subsequent HAT following index admission with alternative pathology; there are clear international guidelines supporting outpatient diagnosis and management of VTE, so genuine cases of HAT may not necessarily require hospital admission or relevant administrative coding; and prior diagnosis of VTE can often be coded during repeat hospital attendance and mistaken as HAT. Many UK hospitals attempting root cause analysis of HAT cases in line with NHS contract standards have developed pathways to mitigate these issues and improve identification, through local reporting arrangements with radiology, ultrasound and pathology services. Such arrangements often work well transiently but are reliant on individuals and reporting systems subject to human error. These issues are reflected in our findings from workstream 2, which report a PPV of 100% for HAT RCA database findings, but limited sensitivity (implying missed cases still occur despite a systematic approach).
More generally, these findings raise questions about the current enthusiasm for data-enabled trials and epidemiological research using outcome measures reliant on routine data methods.162 Such concepts are inherently attractive to researchers and patients, particularly in topic areas with low event rates. However, complex outcome measures which require temporal evaluation and qualification against prior disease states are unlikely to be reliably delivered through use of routinely collected data in isolation. Several explanations have been proposed for current demonstrations of inaccuracy. Relevant data may contain coding errors arising from ambiguous documentation by physicians and inconsistent handling of recurrent cases.162,164 Recent case studies have reported significant amounts of missing data and poor interobserver agreement between routinely collected EHR data accessible through HES and case report form (CRF) evaluation.165 Electronic records contain an abundance of free text, but often lack necessary intelligence to classify patient episodes appropriately, or allow processing and comparison of routinely collected data.166 Recent evidence also suggests that more complex outcomes will be less easily identified through routine coding; a registry study of Medicare claims following mitral valve repair compared to formal adjudication, reported a PPV for mortality of 97%, heart failure requiring hospitalisation of 69%, bleeding of 40% and renal failure of 19%.167 In addition, the time and effort needed to acquire necessary permissions for national routine coding data or to orchestrate data linkage can be substantial. A UK clinical trials unit recently reported a digital request to facilitate routine data comparison to CRFs in the context of a RCT, highlighting a negotiation process over consent that took several years and receipt of data 15 months following application.168 Such time frames may only be realistic within the context of continually adaptive design trials and add further concern as to whether the juice is actually worth the squeeze.
In their call to action, Sydes et al. discuss supplementation of trial-specific follow-up as an option to realise the full potential of data-enabled research.162 Such an approach has potential merit to attempt identification of potential HAT, given the high PPV and high specificity of routine data sources. These cases would require subsequent adjudication and this approach would still fail to address the low sensitivity reported using routine data methods. Our findings suggest the use of routine data sources in isolation would be insufficiently accurate to capture complex VTE and bleeding outcomes within the context of a pragmatic trial context, but supplementation may be a viable use of resources. In addition, routine data sources may have a role in other research contexts, such as identification of cases for qualitative work, case–control studies, targeted individual follow-up or downstream survey work.