Sensitivity and Specificity

The proper use of model evaluation metrics is important for model evaluation and model selection in binary classification tasks. This study investigates how consistent different metrics are at evaluating models across data of different prevalence while the relationships between different variables and the sample size are kept constant. Analyzing 156 data scenarios, 18 model evaluation metrics and five commonly used machine learning models as well as a naive random guess model, I find that evaluation metrics that are less influenced by prevalence offer more consistent evaluation of individual models and more consistent ranking of a set of models. In particular, Area Under the ROC Curve (AUC) which takes all decision thresholds into account when evaluating models has the smallest variance in evaluating individual models and smallest variance in ranking of a set of models. A close threshold analysis using all possible thresholds for all metrics further supports the hypothesis that considering all decision thresholds helps reduce the variance in model evaluation with respect to prevalence change in data. The results have significant implications for model evaluation and model selection in binary classification tasks.

BackgroundAlthough specific biomarkers for primary sclerosing cholangitis (PSC) are required, no such biomarkers have been identified. We previously reported that patients with PSC had anti-integrin αvβ6 autoantibodies at only two hospitals. In this study, we aimed to validate the accuracy of the autoantibodies in diagnosing PSC using the newly developed Anti-integrin αvβ6 enzyme-linked immunosorbent assay (ELISA) Kit, which enables quantitation and comparison of antibodies among different facilities.MethodsOverall, 81 patients with PSC in a Japanese PSC registry recruited from 17 medical centers and hospitals, and 358 controls were enrolled. We retrospectively assessed anti-integrin αvβ6 autoantibodies using the Anti-integrin αvβ6 ELISA Kit and in-house ELISA.ResultsAnti-Integrin αvβ6 ELISA Kit and in-house ELISA exhibited a significant correlation (r = 0.97, P ConclusionsWe validated that anti-integrin αvβ6 autoantibodies have high sensitivity and specificity for diagnosing PSC. This study provides further evidence that anti-integrin αvβ6 autoantibodies are a useful biomarker for diagnosing PSC.

BackgroundDiagnostic delays for tuberculosis are common, with high resultant mortality. Urine-Xpert Ultra (Cepheid) could improve time to diagnosis of tuberculosis disease and rifampicin resistance. We previously reported on lot-to-lot variation of the Fujifilm SILVAMP TB LAM. In this prespecified secondary analysis of the same cohort, we aimed to determine the diagnostic yield and accuracy of Urine-Xpert Ultra for tuberculosis in people with HIV, compared with an extended microbiological reference standard (eMRS) and composite reference standard (CRS) and also compared with Determine TB LAM Ag (AlereLAM, Abbott).MethodsIn this prospective, multicentre, diagnostic accuracy study, we recruited consecutive inpatients and outpatients (aged ≥18 years) with HIV from 13 hospitals and clinics in seven countries (Malawi, South Africa, Tanzania, Thailand, Uganda, Viet Nam, and Zambia). Patients with no isoniazid preventive therapy in the past 6 months and fewer than three doses of tuberculosis treatment in the past 60 days were included. Reference and index testing was performed in real time. The primary outcome of this secondary analysis was the diagnostic yield and accuracy of Urine-Xpert Ultra compared with the eMRS and CRS. Diagnostic accuracy was compared with AlereLAM and diagnostic yield was compared with both AlereLAM and Sputum-Xpert Ultra. This study was registered with ClinicalTrials.gov, NCT04089423, and is complete.FindingsBetween Dec 13, 2019, and Aug 5, 2021, 3528 potentially eligible individuals were screened and 1731 were enrolled, of whom 1602 (92·5%) were classifiable by the eMRS (median age 40 years [IQR 33-48], 838 [52·3%] of 1602 were female, 764 [47·7%] were male, 937 [58·5%] were outpatients, 665 [41·5%] were inpatients, median CD4 count was 374 cells per μL [IQR 138-630], and 254 [15·9%] had microbiologically confirmed tuberculosis). Against eMRS as reference, sensitivities of Urine-Xpert Ultra and AlereLAM were 32·7% (95% CI 27·2-38·7) and 30·7% (25·4-36·6) and specificities were 98·0% (97·1-98·6) and 90·4% (88·7-91·8), respectively. Against CRS as reference, sensitivities of Urine-Xpert Ultra and AlereLAM were 21·1% (95% CI 17·6-25·1), and 30·5% (26·4-34·9), and specificities were 99·1% (98·3-99·6) and 95·1% (93·5-96·3), respectively. The combination of Sputum-Xpert Ultra with AlereLAM or Urine-Xpert Ultra diagnosed 202 (77·1%) and 204 (77·9%) of 262 eMRS-positive participants, respectively, in incompletely overlapping groups; combining all three tests diagnosed 214 (81·7%) of 262 eMRS-positive participants INTERPRETATION: Urine-Xpert Ultra could offer promising clinical utility in addition to AlereLAM and Sputum-Xpert Ultra. In inpatient settings where both AlereLAM and Urine-Xpert Ultra are possible, both should be offered to support rapid diagnosis and treatment.FundingGlobal Health Innovative Technology Fund, KfW Development Bank, Commonwealth of Australia represented by the Department of Foreign Affairs and Trade, and the Netherlands Enterprise Agency.

PurposeOur goal is to develop and validate a practical protocol that guides users in identifying and suppressing electromagnetic noise in low-field MRI systems, enabling operation near the thermal noise limit.MethodsWe present a systematic, stepwise methodology that includes diagnostic measurements, hardware isolation strategies, and good practices for cabling and shielding. Each step is validated with corresponding noise measurements under increasingly complex system configurations, both unloaded and with a human subject present.ResultsNoise levels were monitored through the incremental assembly of a low-field MRI system, revealing key sources of EMI and quantifying their impact. Final configurations achieved noise within 1.5 ×$$ \times $$ the theoretical thermal bound with a subject in the scanner. Image reconstructions illustrate the direct relationship between system noise and image quality.ConclusionThe proposed protocol enables low-field MRI systems to operate close to fundamental noise limits in realistic conditions. The framework also provides actionable guidance for the integration of additional system components, such as gradient drivers and automatic tuning networks, without compromising signal-to-noise ratio (SNR).