Statement of principal findings
In current clinical practice, identification of patients at risk of developing AKI poses a significant challenge to clinicians. Markers of kidney stress and/or injury are hoped to be a useful adjunct to current clinical care, as they may facilitate patient management and informed decisions about treatment. Nevertheless, pathways of presentation and care for AKI are complex, and the potential for modifiability and clinical benefit is uncertain. This assessment looked at the performance of NephroCheck, ARCHITECT and Alinity i urine NGAL assays and BioPorto urine and plasma NGAL assays to assess the risk of AKI in critically ill patients considered for admission to critical care. We included 56 studies with a total of 17,967 participants.
Clinical effectiveness
The main clinical effectiveness findings suggest that these biomarkers may have a role in AKI risk assessment in patients admitted to critical care. Evidence for other clinical settings (cardiac surgery, major non-cardiac surgery) was limited.
The results of meta-analyses indicate that the use of biomarkers may be useful for identifying AKI. However, because of substantial clinical and statistical heterogeneity between studies, and large 95% confidence and prediction regions, there is considerable uncertainty surrounding the validity and reliability of these findings. Moreover, the overall performance of the biomarkers for the detection of AKI, as seen by the meta-analyses of AUC estimates, appears to be modest, rather than excellent, with large boundaries of uncertainty. For example, for the adult population, the highest AUC value for detection of AKI was 0.76, but prediction intervals ranged from 0.33 to 0.99.
For prediction of relevant clinical outcomes, only a small number of studies were available for each biomarker in each clinical setting; this limited the possibility to perform pooled analyses.
Similarly, although there was an indication that addition of biomarkers to existing clinical models might improve the prediction of relevant clinical outcomes, studies varied considerably in terms of study characteristics and statistical methods used to assess prediction, thereby limiting any reliable conclusion.
Overall, as studies varied considerably in terms of clinical setting, timing of sample collection, optimal threshold level, assay platforms, definition of AKI, number of AKI events, time of AKI diagnosis and inclusion/exclusion criteria, the reliability and generalisability of the observed findings are highly uncertain.
We did not find any study that used the Alinity i urine NGAL test or assessed the performance of the biomarkers for prediction of CKD. Similarly, we did not identify any study that assessed the impact of the routine use of the biomarkers on specific clinical outcomes in critically ill patients compared with current standard care.
Cost-effectiveness
A probabilistic decision tree and Markov model were developed (adapted from the model used by Hall et al.97) to describe the care pathway for a mixed prevalence cohort of CKD/no-CKD patients in a hospital setting for patients at risk of developing AKI. The decision tree part of the model captured the acute phase, up to the first 90 days, and modelled the risk of AKI and the potential for the use of biomarkers to prevent AKI or reduce its severity. We used a linked-evidence approach to derive hypothesised links between the presence/absence of AKI and AKI severity on changes in health outcomes (need for ICU care, LOS in hospital, need for acute RRT, 90-day mortality and development of CKD). In the absence of robust trial data, we derived these associations from a large, existing observational data set.102 The Markov model describes the progression of four cohorts (no AKI, AKI 1, AKI 2 and AKI 3) through a set of mutually exclusive health states capturing CKD, ESRD, long-term dialysis, kidney transplant and mortality. Progression through these states depends on an individual’s AKI status in hospital, which influences the starting proportions in the Markov model CKD state.
The model includes health service perspective costs of biomarkers, early application of a KDIGO care bundle, hospitalisation (including ICU and ward costs), acute and long-term dialysis costs, long-term outpatient follow-up costs, and transplant and immunosuppressant costs. Health-state utility values and modelled mortality risk were combined to generate estimates of QALYs gained for each test. The model included the functionality to apply additional follow-up costs and mortality risk over the longer term for patients admitted to the ICU in their index admission.
The cumulative expected value of costs and QALYs were simulated over a lifetime time horizon for each cohort under standard care and each of the biomarker strategies; all results were reported as probabilistic ICERs. We found no trial data that could provide effect estimates for the extent to which biomarkers could both mitigate AKI and improve outcomes. Therefore, the model was built around a series of plausible proportional effects of averting/reducing the severity of AKI on changes in health outcomes. These ranged from optimistic scenarios in which patients who had AKI averted as a result of a biomarker-guided early implementation of a care bundle experienced the same risk of ICU, mortality and CKD as if they were in the no-AKI cohort, to more pessimistic scenarios in which the prevention of AKI or reduction of its severity had no impact on health outcomes.
The costs and QALYs for standard care and for each biomarker test strategy were ranked in ascending order of costs, whereby strategies that were more costly and less effective than an alternative were dominated and excluded from the calculation of the ICERs. In this scenario, the highest ICER under the threshold represents the strategy with the best value for money. All scenarios were also compared directly with standard care. In all cases, the probability of cost-effectiveness from the probabilistic simulation was reported.
Cost-effectiveness results were highly uncertain and ICERs were subject to wide variation depending on the set of scenarios chosen. The probability of cost-effectiveness at an ICER of < £20,000 per QALY gained for scenarios in which NGAL is assumed to be equally as effective as NephroCheck in preventing AKI ranged from 0% to 15% (NephroCheck), 0% to 55% (BioPorto urine NGAL), 0% to 2% (ARCHITECT urine NGAL) and 0% to 48% (BioPorto plasma NGAL). BioPorto urine NGAL was generally the test associated with the greatest probability of cost-effectiveness, albeit this was highly uncertain, when compared with standard care. This is because BioPorto urine NGAL had slightly better diagnostic test accuracy data and slightly lower test costs than the comparator tests. However, there is substantial uncertainty in the diagnostic test accuracy, driven by study heterogeneity; therefore, the results should be interpreted cautiously.
When it is assumed that NGAL tests cannot avert AKI, but can only reduce its severity, the cost-effectiveness case for NephroCheck improves substantially. However, cost-effectiveness remains highly uncertain, with a probability of cost-effectiveness ranging from 0% to 99% across the explored scenarios.
In general, the model results generate a less favourable assessment of cost-effectiveness for the biomarker tests than that of Hall et al.97 There are five reasons for this. First, the prevalence of AKI in the Hall et al.97 study was much higher (31.7%) than in our prevalent AKI population (9.2%). The higher prevalence might be explained by AKI being more common in the ICU setting (starting cohort in Hall et al.97) than in a hospital ward (the starting cohort in our economic model).
Second, the settings are different. Hall et al.97 evaluated the cost-effectiveness of biomarkers for detecting AKI in a critical care setting, whereas our assessment evaluated the cost-effectiveness of AKI biomarkers in a critically ill hospitalised cohort considered for admission to critical care. The data sources used to populate the acute phase of the model are different. Hall et al.97 relied on daily transitions between ICU, hospital and discharge up to 90 days, whereas we have relied on a large observational data set to populate the potential link between changes in AKI status and health outcomes. Therefore, the costs and utilities applied in the acute phase of the base-case models differ between the two analyses.
Third, both models produce estimates of cost-effectiveness that are sensitive to the data used for the diagnostic accuracy of the tests. The diagnostic accuracy data applied in Hall et al.97 are different from those obtained in our meta-analyses, probably because of new studies becoming available since the Hall et al.97 publication and the wider setting for our model. For example, the sensitivity of NephroCheck was 0.90 in Hall et al.97 and 0.75 in our meta-analysis. Consequently, NephroCheck identified more true-positive cases, which generated greater QALY gains in Hall et al.97 than were generated in our model.
Fourth, we take a more conservative approach to the estimation of long-term follow-up costs for the base-case analysis and have not applied excess lifetime costs beyond the 5-year data reported in Lone et al.114
Fifth, we further assume that there is no impact of AKI on follow-up costs beyond the 90 days, whereas Hall et al.97 assume excess costs applied for the full lifetime time horizon. We also assume that the causal impact of AKI on CKD development ceases beyond the first cycle of the Markov model (i.e. 1.25 years after the AKI event), whereas Hall et al.97 assume additional risk of CKD for the full lifetime time horizon of the model.
Overall, both models conclude that there is substantial uncertainty in the results, albeit predicting different base-case ICERs. The results are highly sensitive to key parameters in the model and any combination of the presented scenarios may be plausible.
Strengths and limitations of the assessment
The methods used to conduct this assessment were detailed and thorough. We conducted comprehensive literature searches of major electronic databases and relevant websites and assessed > 1000 full-text studies for eligibility. The large number of screened and extracted articles was necessary because key information (e.g. information on biomarker assays) was not available from the abstracts. This resulted in a need for significant literature screening resources and for considering strict inclusion criteria to ensure that the assessment remained feasible and timely. We restricted inclusion to studies that enrolled at least 100 participants and excluded studies on low-birthweight and preterm babies. It is possible that inclusion of all existing studies, irrespective of the sample size, might produce relevant findings. However, we reached a consensus that small and niche studies would not provide clinically generalisable evidence for pooling and would be underpowered to provide reliable evidence in isolation. Low-birthweight and preterm babies were considered a category of patients with specific care needs that are not generalisable to the population included in this assessment.
The primary weakness of the systematic review of clinical effectiveness evidence was the substantial clinical heterogeneity observed between studies. There was considerable heterogeneity, especially with regard to NGAL threshold levels, time of sample collection, definition of AKI and prevalence of AKI, time of AKI diagnosis, and assay platforms. Consequently, the diagnostic accuracy of individual tests varied considerably and the confidence and prediction regions in the pooled analyses were notably large. Moreover, when the studies had smaller numbers of AKI events (low prevalence), the relationship observed between sensitivity and specificity estimates became quite different from that of studies for which prevalence was higher.
Indeed, the shape and size of the prediction regions in the SROC plots were influenced by studies that showed a different relationship between sensitivity and specificity, compared with other studies. Hence, we do not have much confidence in the pooled estimates.
In particular, the intrinsic complexity of this assessment (multiple research questions, multiple biomarkers and sample media, multiple clinical settings, broad patient population, differences in assay platforms, definition of AKI) means that the findings reported here are also complex, particularly given the absence of robust trial evidence to support economic model development. Although the original scope of this assessment was the assessment of hospitalised patients considered to be at risk of admission to critical care, no studies focused on this specific group of patients (pre admission to critical care). Most studies were conducted outside the UK and assessed patients already admitted to intensive or critical care after different surgical procedures or with different (or multiple) clinical conditions. Furthermore, the provision of intensive care resources across the world are heterogeneous, so many studies will not be representative of how intensive care is utilised in the UK. This means that it is unclear how well the findings of heterogeneous studies which are predominantly based in intensive care and conducted outside the UK can be applied to a UK clinical scenario of people not currently receiving critical care, but at risk of requiring it.
Criteria used for the definition of AKI were consistent with current KDIGO recommendations, but differed slightly across studies with respect to operationalisation. This means that the extent of bidirectional misclassification of AKI and CKD may vary between studies and setting, which may affect biomarker performance.143 In some studies, it was unclear whether or not the reported associations between biomarkers and AKI were indeed attributed to kidney injury. The current definition of AKI is based on elevations in serum creatinine concentration, which poses the conundrum of using an imperfect standard to assess the performance of biomarkers. Serum creatinine is not always measured at the same frequency as biomarkers are measured, and to ascertain the exact time of creatinine rise is problematic. As a result, the ‘ground truth’ of AKI existence could not be established with a gold-standard reference in any of the studies. In addition, no studies considered alternative methods for early or incipient AKI detection, such as the use of machine learning algorithms.144
In some studies, we observed a very small number of AKI events, compared with other included studies. Interestingly, two studies conducted in the cardiac surgery setting [a medium-size single-centre study26 with 400 participants and a large multicentre study,38 the Translational Research Investigating Biomarker Endpoints (TRIBE) trial, with 1219 participants)] showed similar prevalence rates (4% and 5%, respectively) and a similar pattern of accuracy (poor sensitivity estimates and good specificity estimates). The number of AKI events are known to vary depending on both AKI definition and clinical setting, which underlies the heterogeneity of existing studies.
An unavoidable limitation of this evaluation is the variation in the use of NGAL tests. Threshold cut-off points to classify patients with and patients without AKI in each clinical setting were not consistent across studies. This means that differences between studies could relate to the chosen threshold, rather than NGAL performance. We selected one threshold per study, according to our inclusion criteria, and estimated the underlying SROC curve using a hierarchical model, which takes into account the within- and between-study variability. NGAL studies also varied with respect to analytic methods of measurement. Some studies used absolute urine concentrations, whereas others used NGAL concentrations normalised for urine creatinine concentrations. There were insufficient data available per type of biomarker and clinical setting to further investigate this source of variability, and determine the extent to which analytic methods influence estimates of diagnostic accuracy and whether or not it was sensible to pool results across studies. Nevertheless, we note that in the multicentre TRIBE prospective study38 assessing 1219 adults undergoing cardiac surgery, the authors repeated the analyses using NGAL urine-creatinine corrected values and did not observe improvements in the AUC values, compared with uncorrected results.
Several studies did not provide sensitivity, specificity and AUC values for the biomarkers for the diagnostic or prognostic accuracy of AKI. In future studies, accuracy measures such as sensitivity and specificity must be considered and defined rigorously at transparent cut-off points for predictive biomarkers, as they may need to vary according to clinical setting.145
Notwithstanding analytic and threshold heterogeneity, the number of available studies for each type of assay in each clinical setting limited our ability to assess the role of the biomarkers for the prediction of relevant clinical outcomes. Furthermore, the number of events was small in many studies and the duration of follow-up was not consistent across studies, so mortality and RRT could not be reliably assessed at the same time points. Furthermore, details of the methods used for prediction analyses were insufficient in many studies. Although information on adjustment strategies and on the process of variable selection were usually provided, the original cohort of potential predictors, prior to the multivariable analysis, was never clearly specified, leading to potential risk of bias.
Finally, introduction of a biomarker would require evidence not just that it performs well as a predictor of modifiable and intervenable AKI, but also that there is incremental improvement of existing or alternative approaches to clinical care. There was insufficient information to determine with certainty whether or not the biomarkers had an incremental advantage over the traditional marker of serum creatinine and urine output, or had information available for clinical assessment. Only a limited number of studies compared the AUC of the biomarkers under investigation with that of serum creatinine for the detection of AKI, and fewer studies compared the performance of the biomarkers with that of clinical models for prediction of AKI or of relevant patient outcomes.
Uncertainties
Clinical effectiveness evidence
There is considerable uncertainty surrounding the generalisability of the studies to the UK population. Most of the studies were conducted outside the UK and assessed patients already admitted to critical care. Because no studies were identified for inclusion, we were not able to assess the impact that the routine use of these biomarkers may have on clinical outcomes of critically ill people considered for admission to critical care, compared with standard clinical assessment.
At present, in the literature, there is limited information on the benefits of incorporating biomarker results with those of current clinical criteria (serum creatinine and urine output) to improve the clinical management of patients with AKI. Recently, Zarbock et al.,146 in a RCT of critically ill surgical patients with AKI, assessed the use of early versus delayed RRT. Plasma NGAL of > 150 ng/ml was one of the inclusion criteria, together with the KDIGO criteria. The trial results showed that early RRT, compared with delayed RRT, reduced mortality, duration of RRT and hospital stay, and that the combination of the KDIGO classification system with plasma NGAL was effective in identifying patients with deteriorating AKI. Subsequent negative results from the Artificial Kidney Initiation in Kidney Injury (AKIKI) RCT147 for critically ill medical patients suggest that these findings may apply to targeted circumstances only.
More recently (in 2017), the Zarbock group conducted a biomarker-guided RCT of patients who underwent cardiac surgery.110 They used a biomarker-based approach (NephroCheck test) to identify high-risk patients and implement a bundle of supportive measures recommended by the KDIGO guidelines to reduce the occurrence of AKI, as well as that of mortality and RRT. Their results showed that implementation of the KDIGO guidelines, compared with standard care, reduced the frequency of AKI in the 72 hours after cardiac surgery. However, the trial did not show a reduction in the need for RRT, an improvement in mortality or a positive effect measure on any hard clinical outcome. The authors concluded that future, adequately powered, multicentre trials are required. Similarly, Göcze et al.,116 in a study of major non-cardiac surgery patients, showed that the early adoption of a bundle of supportive measures according to the KDIGO guidelines in patients with NephroCheck concentrations of > 0.3 (ng/ml)2/1000 resulted in a reduced occurrence of AKI, decreased hospital and ICU stay, and reduced costs, but, again, there was no evidence of improvement of hard outcomes (need for RRT, mortality, or major kidney events).
Overall, despite some evidence suggesting possible improvement of care processes and health care use when biomarker-guided care bundles are used alongside KDIGO criteria, there is still considerable uncertainty regarding effects on health outcomes, particularly when used in the pre-critical care setting. In addition, the optimal threshold for NGAL and how this changes according to different clinical settings have yet to be established. Future studies should evaluate the targeted use of the biomarkers within specific clinical populations and circumstances in which there is potential for benefit with a plausible and feasible intervention. In particular, they should focus on the assessment of the impact of routine biomarker use on a reduction in mortality, major clinical adverse events, modification of clinical care, and resource use. In other words, future research should evaluate the use of these biomarkers to improve patients’ clinical outcomes and management.
Discrete urine and plasma NGAL cut-off points for differentiating between AKI and non-AKI patients in each clinical setting need to be identified and the timing of collection of biomarker concentrations should be set out more clearly according to each setting. In line with the recommendations from the 10th Acute Dialysis Quality Initiative consensus conference,148 there is also a need to harmonise the methods and platforms for collection, handling and storage of urine and plasma samples. Furthermore, it would be useful to harmonise the reporting of biomarker concentrations (e.g. absolute concentrations, ratio to urine creatinine) and corroborate techniques for normalising urine biomarker concentrations to urine creatinine concentrations.
Finally, it is well recognised that AKI encompasses a range of clinical aetiologies, phenotypes and patterns of renal recovery. In addition, current measures of AKI may be insufficient to disentangle AKI that is predominantly functional without kidney damage from people with incipient subclinical damage, and people with both AKI and kidney damage. In this context, it remains unclear how phenotypic information on people with AKI should most usefully be combined to help target those most likely to benefit from earlier recognition and timely intervention, nor how such an intervention may differ between clinical phenotypes.148
Cost-effectiveness evidence
There are three key areas of uncertainty in the economic evaluation modelling that limit the robustness of the cost-effectiveness results: (1) the lack of evidence on the impact of the biomarkers on health outcomes, (2) the heterogeneity in the diagnostic accuracy data (including uncertainty in the prevalence of AKI in a broad, poorly defined population) and (3) the uncertainty around the impact of a NGAL-guided implementation of a KDIGO care bundle on the frequency and severity of AKI. Given these uncertainties, the choice of a preferred base-case scenario is challenging and the observed results should be considered cautiously. These are speculative analyses ranging from a set of pessimistic scenarios to a set of optimistic scenarios for the use of the biomarkers under assessment.
Specifically, there is no evidence to describe the impact of the use of the AKI biomarkers on important health outcomes (such as need for ICU care, length of hospital stay, risk of 90-day mortality or development of new/progression of existing CKD). Accordingly, the cost-effectiveness results are based on a linked-evidence approach, whereby we have relied on observational associations to infer how prevention or mitigation of AKI may affect changes in health outcomes. These associations necessitate causal assumptions, but, although a causal link between AKI and poor outcomes is plausible, the extent of this causal relationship is uncertain and controversial.149,150 The cost-effectiveness results are therefore presented for a range of alternative, but potentially plausible, scenario analyses ranging from a set of optimistic assumptions in which biomarker-guided care bundles may lead to substantial improvements in health outcomes (need for ICU care, CKD, mortality) to a set of more conservative assumptions in which change in AKI status has no effect on health outcomes. It is likely that the true estimate of cost-effectiveness lies somewhere between these two extremes.
Furthermore, the diagnostic accuracy data used in the economic model are obtained from studies that are considerably heterogeneous in terms of baseline AKI prevalence, timing of sample collection, threshold values and definition of AKI. Given the difficulty in defining the population that fits within the scope of this assessment, it is unclear how generalisable the diagnostic accuracy data are to the UK population in which the biomarkers could be used.
Also of note are additional uncertainties in the model that make it difficult to come to conclusions about the relative cost-effectiveness of each biomarker. For example, although there is some evidence in the literature, from Meersch et al.,110 that early NephroCheck-guided implementation of a KDIGO care bundle may improve AKI status at 72 hours, the potential for similar improvements using NGAL is unknown. Therefore, we have considered two scenarios for the cost-effectiveness analyses. The first assumes, optimistically, that all NGAL tests are equally as effective at preventing AKI or reducing its severity as NephroCheck; the second, based on the available data from Meersch et al.,110 assumes that NGAL can reduce the severity of AKI once it occurs, but cannot prevent its occurrence.
Because of these uncertainties, the results of the cost-effectiveness modelling are largely speculative and should be interpreted with caution. Although extensive probabilistic analyses are carried out for scenario analyses, these may still not fully capture the uncertainty faced in the implementation of these biomarkers in clinical practice.
In summary, the current evidence base is insufficient to make a full appraisal of the economic value of the biomarkers under investigation to provide cost-effective improvements in clinical outcomes of AKI. Therefore, we have provided a range of scenarios that cannot answer the full remit of this evaluation. We believe that the scenarios illustrate what might be required for the biomarkers to be cost-effective, highlighting, through the assumptions involved, the current gaps where further research is required.