Calibration and further development of the cardiovascular disease microsimulation model

Aims and objectives

In this chapter, we discuss the calibration, further development and validation of the CVD microsimulation model in the UKB cohort and the Whitehall II study. Although CTTC trial data included well-adjudicated CVD end points, the model needs to represent well the contemporary UK CVD and mortality risks. Therefore, our aim was to use the large and contemporary UKB cohort to check the initial model performance and generate a reliable CVD microsimulation model capable of predicting CVD risks and mortality in different categories of the current UK population based on sociodemographic and clinical risk factors.

Methods

UK Biobank cohort

Between 2006 and 2010, the UKB cohort study recruited over 500,000 individuals aged 40–70 years from across the UK.31 Study participants were followed using routine electronic healthcare databases to identify incidences of deaths, cancers and hospital admissions. Additionally, approximately 40% of the participants had their primary care records linked to the study. The UKB database contains comprehensive information on the sociodemographic, behavioural, physical and clinical characteristics of the participants at recruitment, as well as their disease histories both at recruitment and during follow-up. All UKB participants were included in the present study, with the exception of a small number of individuals with end-stage kidney disease at baseline. Appendix 1 describes the processing of UKB data to support CVD model development and QRISK3 calculation21 (see Appendix 1, Table 22).

Cardiovascular disease model development in UK Biobank

We utilised the follow-up data from the UKB up to 31 March 2017 to validate, calibrate and further develop the event risk equations for MI, stroke, CRV, incident cancer, VD and NVD. About 9500 participants experienced MI, 5000 experienced strokes and 3400 died from CVD causes, indicating sufficient sample sizes for the development of risk equations. During the external validation of the initial microsimulation model in the UKB database, we found that all risk equations required calibration.

The general approach to calibration and further development of the risk equations involved four steps. First, we calibrated the intercept and shape parameters of all trial data-based initial risk equations using the calculated linear predictor, which was calculated using initial risk equation coefficients and UKB participant characteristics. Second, we re-estimated the coefficients of covariates with different definitions in UKB (e.g. smoking and ethnicity categories and diabetes and cancer during follow-up) and related interaction terms by releasing them from the linear predictor. Third, further individual participant covariates of interest, such as physical activity, socioeconomic deprivation, diet quality, history of severe mental illness and type 1 diabetes, which were not available in CTTC data, were included in the risk equations. Finally, we fitted an incident diabetes risk equation with glycated haemoglobin (HbA1c) at entry as a further covariate using data from UKB participants with linked primary care records. The incident diabetes risk equation was estimated across both participants with and participants without previous CVD as there was no external evidence of difference in associations with risk factors, the PH assumptions were met and there were no substantive differences in estimated associations in the two populations. The incident diabetes equation was integrated as an additional end point in the microsimulation model.

Cox regression models were used for backward and forward stepwise covariate selection, with an inclusion threshold of 1% level of statistical significance. PH assumption of covariates in each model was tested using Schoenfeld residuals. Parametric PH models were then estimated for each end point, using three distributions (exponential, Weibull and Gompertz), and their Akaike information criterion (AIC) and Bayesian information criterion (BIC) were compared to identify the distributions best fitting each model end point. Lifetime simulations were used for the end points of death, incident cancer and incident diabetes to assess whether the model predicted lifetime risks were consistent with other UK population data. To account for parameter uncertainty, we generated 1000 sets of coefficients using bootstrapping (with replacement) of the respective UKB data and re-estimating the final risk equations.36

Executing model predictions

We executed 1000 first-order simulations for each individual and used diagnostics to determine the sufficient number of microsimulations across study results. A simulation stops when the individual is predicted to die or reaches 110 years of age. We summarised risks of model events, survival and quality-adjusted life-years (QALYs) across the first-order simulations to minimise the Monte Carlo uncertainty in the model projection for each participant.

We employed the CVD microsimulation model to perform lifelong projections for all UKB participants. Individuals with cancer and/or diabetes history at entry were not at risk of incident cancer or diabetes. The lifetime risks of model events, remaining life expectancy and QALYs for each participant were derived by adding up the results across annual cycles.

We also ran 500 and 1000 simulations for probabilistic sensitivity analysis in participant categories without and with prior CVD, respectively, and checked that the 95% confidence intervals (CIs) for cost-effectiveness measures were reliably estimated.

Further model validation in UK Biobank and Whitehall II studies

Cumulative end-point incidences predicted by the UKB-calibrated CVD microsimulation model were compared with the observed cumulative end-point incidences across UKB participants by previous CVD, sex and age at entry over the 9 years of UKB data used in the model calibration (until 31 March 2017; internal validation) and over further 3 years of UKB follow-up data not used in model estimation (until 29 February 2020; temporally external validation). The UKB-calibrated model was further externally validated in the Whitehall II cohort32 using the Phase 9 data of participants as baseline and comparing the model-predicted with the observed cumulative incidences of end points over the next 10 years of follow-up. Appendix 2 describes the Whitehall II data used in CVD model validation, including participants’ baseline characteristics (see Appendix 2, Table 23) and the number of events they experienced during follow-up (see Appendix 2, Table 24).

Results

The analytical data from the UKB study included 444,576 participants without previous CVD and 57,278 with previous CVD at entry. The average follow-up time was 8.1 years for participants without previous CVD and 7.9 years for those with previous CVD, resulting in 3,588,967 and 451,368 person-years of follow-up, respectively. At entry, the mean age of participants without previous CVD was 56 [standard deviation (SD) 8] years, with 56% being female, 95% being of white ethnicity, 5% having diabetes and 16% receiving hypertension treatment. The mean age of participants with previous CVD at entry was 60 (SD 7) years, and 41% were female, 95% were of white ethnicity, 14% had diabetes and 46% were receiving hypertension treatment (Table 4).

The numbers of events over follow-up that informed the estimation and calibration of model risk equations are summarised in Table 5.

Risk equations

The external validation of the initial microsimulation model (see Chapter 2) in the UKB cohort showed that the predicted cumulative incidence rates of CVD events were significantly higher than the observed incidence rates, indicating a need for model calibration to enhance the accuracy of predictions in this UK cohort.

The final parametric PH models, following the four steps of calibration and further development (see Appendix 3, Tables 25–27), are presented in Table 6 (type of parametric model and shape parameters with 95% CI) and Figure 4 [hazard ratio (HR) (95% CI)].

FIGURE 4 Hazard ratios (95% CI) of risk factors in final risk equations. (a) MI; (b) stroke; (c) CRV; (d) incident diabetes; (e) incident cancer; (f) VD; and (g) NVD. (Figure not reproduced here; see the original article.)

Age, male sex, smoking and treated hypertension were strongly associated with higher risks of CVD events. Higher socioeconomic deprivation was associated with higher risks of stroke, incident diabetes, VD and NVD. An unhealthy diet was associated with higher risks of MI, stroke and VD among people without CVD history. Lower physical activity was associated with higher risk of MI in people without CVD history; higher risk of stroke in people with CVD history; and higher risks of cancer, diabetes and NVD regardless of CVD history (see Figure 4).

MI, especially recent MI, was strongly associated with higher risk of CRV, and, separately, higher risk of stroke. MI and stroke were associated with higher risks of subsequent VD, with the greatest risks in the year of event. CRV was associated with lower risk of subsequent VD. Longer time since diabetes diagnosis and higher HbA1c level, in those without diabetes, were associated with higher risks of all CVD events. The patterns were similar in people without and people with CVD history, although magnitudes differed (see Figure 4). See Appendix 3 for the detailed risk equations (see Appendix 3, Table 28).

The diagnostics for the number of first-order simulations (see Appendix 4) indicated that 500 first-order simulations achieved stable estimates across the participant categories of interest (see Appendix 4, Figure 14). The diagnostics for the number of simulations for the probabilistic sensitivity analyses indicated that 500 and 1000 simulations in people without and people with previous CVD, respectively, were sufficient to achieve stable estimates of the 95% CIs (see Appendix 4, Figure 15).

After calibration, the model’s predicted cumulative incidence rates for all end points across follow-up years showed good agreement with the observed rates in participant categories by previous CVD, age, sex and prior diabetes in the UKB (Figures 5a and 6). The UKB-calibrated model also demonstrated good overall performance in the external validation in the Whitehall II cohort, with only a slight overestimation of stroke risk among participants without previous CVD (see Figure 5b).

FIGURE 5 Validation of the CVD model in UKB and Whitehall II cohorts. (a) Validation in UKB; and (b) validation in Whitehall II Phase 9. Red, observed; black, model. Validation covers 12 years in UKB (8 years for incidence diabetes due to stopping follow-up earlier) and 10 years in Whitehall II (7 years for incidence cancer due to stopping follow-up earlier). (Figure not reproduced here; see the original article.)

FIGURE 6 Validation of the CVD model in UKB cohort, by sex, age and prior diabetes at entry. Red, observed; black, model. Validation includes 3 extra years that were not used to calibrate the model. Follow-up of incident diabetes partly relies on primary care records, which ended earlier than other data types in the UKB. Incident diabetes is not predicted for individuals with diabetes history. (Figure not reproduced here; see the original article.)

Model prediction in UK Biobank

We used the calibrated model to execute individual projections for all UKB participants. For individuals in the same age and sex category, shorter life expectancy and fewer QALYs were predicted for those with CVD history or higher 10-year CVD risk. Men had shorter life expectancy but more QALYs as a proportion of their life expectancy than women, reflecting higher QoL in men all else equal. Taking participants aged 50–59 years as an example, the projected remaining life-years ranged between 24.2 (95% CI 23.1 to 25.5) and 34.6 (33.5 to 35.8) years for men [15.7 (15.1 to 16.5) to 29.4 (28.5 to 30.3) QALYs] and between 28.0 (27.7 to 30.1) and 37.2 (36.0 to 38.5) years for women [16.4 (16.0 to 17.4) to 29.0 (28.0 to 29.8) QALYs], depending on their CVD history or 10-year CVD risk (see the summary of model predictions and their parameter uncertainty presented in Appendix 5, Table 29).

Summary

The CVD microsimulation policy model, which was calibrated in the UKB cohort, demonstrated good performance across different categories of UKB participants and Whitehall II participants. The model can be used to project individuals’ lifetime risks of cardiovascular morbidity, incident diabetes, incident cancer, and vascular and nonvascular mortality, and long-term effects of strategies to reduce CVD risks.

Report Supplementary Material 1 includes the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD) Checklist for the Prediction Model Development and Validation.