Area Under Curve

BackgroundROC (receiver operating characteristic) curve analysis is well established for assessing how well a marker is capable of discriminating between individuals who experience disease onset and individuals who do not. The classical (standard) approach of ROC curve analysis considers event (disease) status and marker value for an individual as fixed over time, however in practice, both the disease status and marker value change over time. Individuals who are disease-free earlier may develop the disease later due to longer study follow-up, and also their marker value may change from baseline during follow-up. Thus, an ROC curve as a function of time is more appropriate. However, many researchers still use the standard ROC curve approach to determine the marker capability ignoring the time dependency of the disease status or the marker.MethodsWe comprehensively review currently proposed methodologies of time-dependent ROC curves which use single or longitudinal marker measurements, aiming to provide clarity in each methodology, identify software tools to carry out such analysis in practice and illustrate several applications of the methodology. We have also extended some methods to incorporate a longitudinal marker and illustrated the methodologies using a sequential dataset from the Mayo Clinic trial in primary biliary cirrhosis (PBC) of the liver.ResultsFrom our methodological review, we have identified 18 estimation methods of time-dependent ROC curve analyses for censored event times and three other methods can only deal with non-censored event times. Despite the considerable numbers of estimation methods, applications of the methodology in clinical studies are still lacking.ConclusionsThe value of time-dependent ROC curve methods has been re-established. We have illustrated the methods in practice using currently available software and made some recommendations for future research.

Genome-wide association studies in human populations have facilitated the creation of genomic profiles which combine the effects of many associated genetic variants to predict risk of disease. The area under the receiver operator characteristic (ROC) curve is a well established measure for determining the efficacy of tests in correctly classifying diseased and non-diseased individuals. We use quantitative genetics theory to provide insight into the genetic interpretation of the area under the ROC curve (AUC) when the test classifier is a predictor of genetic risk. Even when the proportion of genetic variance explained by the test is 100%, there is a maximum value for AUC that depends on the genetic epidemiology of the disease, i.e. either the sibling recurrence risk or heritability and disease prevalence. We derive an equation relating maximum AUC to heritability and disease prevalence. The expression can be reversed to calculate the proportion of genetic variance explained given AUC, disease prevalence, and heritability. We use published estimates of disease prevalence and sibling recurrence risk for 17 complex genetic diseases to calculate the proportion of genetic variance that a test must explain to achieve AUC = 0.75; this varied from 0.10 to 0.74. We provide a genetic interpretation of AUC for use with predictors of genetic risk based on genomic profiles. We provide a strategy to estimate proportion of genetic variance explained on the liability scale from estimates of AUC, disease prevalence, and heritability (or sibling recurrence risk) available as an online calculator.

Precision-recall curves are highly informative about the performance of binary classifiers, and the area under these curves is a popular scalar performance measure for comparing different classifiers. However, for many applications class labels are not provided with absolute certainty, but with some degree of confidence, often reflected by weights or soft labels assigned to data points. Computing the area under the precision-recall curve requires interpolating between adjacent supporting points, but previous interpolation schemes are not directly applicable to weighted data. Hence, even in cases where weights were available, they had to be neglected for assessing classifiers using precision-recall curves. Here, we propose an interpolation for precision-recall curves that can also be used for weighted data, and we derive conditions for classification scores yielding the maximum and minimum area under the precision-recall curve. We investigate accordances and differences of the proposed interpolation and previous ones, and we demonstrate that taking into account existing weights of test data is important for the comparison of classifiers.

BACKGROUND: Sample size determination for area under the curve (AUC) comparisons in diagnostic accuracy studies requires the consideration of multiple methodological parameters. The type of diagnostic test, the nature of the data (discrete or continuous), the correlation structure between tests, and the degree of AUC differences all influence optimal study design and planning. To address these factors, this study provides comprehensive sample size and power calculations for comparing AUCs between diagnostic tests across clinically relevant scenarios. METHODS: We conducted a comprehensive evaluation of sample size and power analysis for AUC comparisons under varying correlation levels (ρ = 0.30, 0.50, 0.80), data types (discrete vs. continuous), and AUC differences (ΔAUC = 0.02–0.10). The Obuchowski–McClish method was applied for discrete data, and the Hanley–McNeil approach was applied for continuous data, assuming balanced case-control allocation and two-sided testing (α = 0.05). Sample sizes were calculated to achieve 80% statistical power based on established methodological approaches and adjusted for a 20% anticipated attrition rate. Power curves were generated to illustrate the relationship between sample size and statistical power across the evaluated scenarios. RESULTS: The required sample sizes varied substantially across scenarios, ranging from 36 − 3,709 participants per group to achieve 80% power. The degree of difference in the area under the curve (AUC) was the primary determinant: ΔAUC = 0.02 required 909-3,709 participants, whereas a ΔAUC = 0.10 required only 36–142 participants. Inter-test correlation also had a marked impact on efficiency, with a strong correlation (ρ = 0.8) reducing sample sizes by 49% in discrete models and 68% in continuous models compared with a weak correlation (ρ = 0.3). Continuous data models consistently outperformed discrete models, requiring 24–53% fewer participants across all the scenarios. The most demanding scenario (ΔAUC = 0.02, discrete data, ρ = 0.3) required 3,709 participants per group, whereas the most efficient scenario (ΔAUC = 0.10, continuous data, ρ = 0.8) required only 36 participants, representing a 103-fold difference. CONCLUSIONS: Methodological choices lead to substantial variations in sample size requirements for diagnostic accuracy studies. Optimal parameter selection, particularly the use of continuous data models and accounting for strong inter-test correlations, can reduce the required sample sizes by up to 68% compared with suboptimal combinations. These results provide evidence-based guidance for efficient planning of diagnostic accuracy studies and underscore the critical importance of methodological considerations in study design optimization.