Introduction
This chapter describes the development and internal validation of a model to predict the risk of re-dislocation. Prediction models using routinely collected data from primary care were developed in the surgical and non-surgical cohorts separately, using the CPRD-HES linked data set.
Methods
Sample
The collated sample comprised patients aged 16–35 years with a first-time TASD who had at least 2 years of data (washout period) before a first-time entry Read code for shoulder dislocation and with up to 3 years of follow-up coding, who were registered at ‘active’ general practices in England (see Figure 11).
Definition of the primary outcome
The primary outcome was re-dislocation following a first-time TASD. We determined an entry date for each participant, which was the date of surgery for the surgical cohort and the date of first shoulder dislocation for the non-surgical cohort. Observation time was calculated from the entry date to an exit date, which was defined as the earliest date of recorded re-dislocation or 3 years after the index date.
Candidate predictors
Potential predictors of re-dislocation were defined a priori by expert consensus and informed by the validation study (see Appendix 4). Only eight of these predictors were available in the CPRD and were used as candidate predictors in the multivariable prediction model (Table 22). Owing to the sparseness of the CCI score in the surgical cohort, it was not considered in the model building in this cohort.
| Candidate predictors (surgical patients) | Candidate predictors (non-surgical patients) |
|---|---|
| Age (years) | Age (years) |
| Sex | Sex |
| BMI (kg/m2) | BMI (kg/m2) |
| Smoking status (non-smoker, current smoker, ex-smoker) | Smoking status (non-smoker, current smoker, ex-smoker) |
| Alcohol consumption (reference: non-drinker) | Alcohol consumption (reference: non-drinker) |
| Epilepsy | Epilepsy |
| Painkillers within 1 month of index date | Painkillers within 3 months of index date |
| IMD | IMD |
| CCI score |
Continuous predictors
Fractional polynomials were used to explore the presence of non-linear relationships of continuous predictors (e.g. age, BMI); however, a linear relationship was found to be a good approximation.34
Missing data
We assumed that missing data occurred at random and we carried out multiple imputation.24 Missing values were predicted on the basis of all other predictors as well as the outcome. One hundred imputed data sets were generated with imputed values, reflecting the uncertainty associated with the imputations. Models were fitted on each imputed data set and coefficients combined using Rubin’s rules.
Model development
All candidate predictors in Table 22 were included in the multivariable Cox regression models for predicting re-dislocation. Because predictors were chosen a priori based on clinical consensus, and only a small number of candidate predictors were available in the CPRD, no reduction of predictors was considered.
Assessment of model performance and internal validation
The predictive ability of the model was assessed in terms of discrimination.35 Discrimination is the ability of the model to differentiate between individuals who have a re-dislocation and those who do not. Discrimination was assessed by calculating the concordance (c)-index; a value of 0.5 indicates no discrimination (equivalent to tossing a coin) and a value of 1 indicates perfect discrimination.
Optimism in the performance was assessed by bootstrap resampling.35 We drew 200 samples with replacement from the original data, with the same size as the original derivation data. In each bootstrap sample the entire modelling process was repeated. This process was repeated over each of the 100 imputed data sets, and an averaged, optimism-corrected c-index was taken.
The R software environment (version 3.5.0) was used for all analyses. We followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis statement for reporting our analyses.36,37
Results
The characteristics of the surgical and non-surgical groups are described in Table 23. There were 342 eligible individuals in the surgical cohort, of which 61 (18%) went on to have a re-dislocation within 3 years of the index date. There were 4271 individuals in the non-surgical cohort, of which 851 (20%) went on to have a re-dislocation within the study period. There were large numbers of missing data, most notably for BMI (46% missing in the surgical cohort and 42% missing in the non-surgical cohort) and alcohol consumption (51% missing in the surgical cohort and 43% missing in the non-surgical cohort). Forty-one per cent (n = 140) of individuals have no missing information on all eight predictors for the model developed in the surgical cohort, and 46% (n = 1960) have no missing information on all nine predictors for the model developed in the non-surgical cohort.
| Variable | Patient group, n (%) | Total, n (%) | |||
|---|---|---|---|---|---|
| No surgery | Surgery | ||||
| No re-dislocation | Re-dislocation | No re-dislocation | Re-dislocation | ||
| Total number of participants | 3420 (80.1) | 851 (19.9) | 281 (82.2) | 61 (17.8) | 4613 |
| Year of shoulder dislocation | |||||
| 1997 | 54 (1.6) | 10 (1.2) | 1 (0.4) | 0 (0.0) | 65 (1.4) |
| 1998 | 82 (2.4) | 27 (3.2) | 5 (1.8) | 1 (1.6) | 115 (2.5) |
| 1999 | 101 (3.0) | 24 (2.8) | 2 (0.7) | 1 (1.6) | 128 (2.8) |
| 2000 | 127 (3.7) | 28 (3.3) | 5 (1.8) | 1 (1.6) | 161 (3.5) |
| 2001 | 153 (4.5) | 46 (5.4) | 7 (2.5) | 3 (4.9) | 209 (4.5) |
| 2002 | 189 (5.5) | 50 (5.9) | 10 (3.6) | 1 (1.6) | 250 (5.4) |
| 2003 | 234 (6.8) | 54 (6.3) | 17 (6.0) | 4 (6.6) | 309 (6.7) |
| 2004 | 232 (6.8) | 58 (6.8) | 10 (3.6) | 1 (1.6) | 301 (6.5) |
| 2005 | 213 (6.2) | 58 (6.8) | 13 (4.6) | 4 (6.6) | 288 (6.2) |
| 2006 | 230 (6.7) | 62 (7.3) | 21 (7.5) | 2 (3.3) | 315 (6.8) |
| 2007 | 240 (7.0) | 68 (8.0) | 20 (7.1) | 6 (9.8) | 334 (7.2) |
| 2008 | 239 (7.0) | 67 (7.9) | 28 (10.0) | 5 (8.2) | 339 (7.3) |
| 2009 | 238 (7.0) | 65 (7.6) | 20 (7.1) | 8 (13.1) | 331 (7.2) |
| 2010 | 244 (7.1) | 70 (8.2) | 29 (10.3) | 5 (8.2) | 348 (7.5) |
| 2011 | 251 (7.3) | 50 (5.9) | 24 (8.5) | 4 (6.6) | 329 (7.1) |
| 2012 | 207 (6.1) | 54 (6.3) | 26 (9.3) | 6 (9.8) | 293 (6.4) |
| 2013 | 170 (5.0) | 39 (4.6) | 22 (7.8) | 6 (9.8) | 237 (5.1) |
| 2014 | 174 (5.1) | 21 (2.5) | 19 (6.8) | 3 (4.9) | 217 (4.7) |
| 2015 | 42 (1.2) | 0 (0.0) | 2 (0.7) | 0 (0.0) | 44 (1.0) |
| Age (years), median (IQR) | 24 (20–29) | 22 (19–26) | 24 (20–28) | 21 (19–23) | 23 (19–28) |
| Sex | |||||
| Male | 2772 (81.1) | 720 (84.6) | 250 (89.0) | 52 (85.2) | 3794 (82.2) |
| Female | 648 (18.9) | 131 (18.9) | 31 (11.0) | 9 (14.8) | 819 (17.8) |
| BMI (kg/m2) | |||||
| < 25 | 1132 (33.1) | 307 (36.1) | 87 (31.0) | 16 (26.2) | 1542 (33.4) |
| 25.0–29.9 | 584 (17.1) | 124 (14.6) | 46 (16.4) | 9 (14.8) | 753 (16.5) |
| ≥ 30.0 | 277 (8.1) | 61 (7.2) | 24 (8.5) | 2 (3.3) | 364 (7.9) |
| Missing | 1427 (41.7) | 359 (42.2) | 124 (44.1) | 34 (55.7) | 1944 (42.1) |
| IMD 2004 (quintile of deprivation) | |||||
| 1 (affluent) | 948 (27.7) | 225 (26.4) | 62 (22.1) | 6 (9.8) | 1241 (26.9) |
| 2 | 748 (21.9) | 201 (23.6) | 65 (23.1) | 14 (23.0) | 1028 (22.3) |
| 3 | 688 (20.1) | 158 (18.6) | 52 (18.5) | 17 (27.9) | 915 (19.8) |
| 4 | 602 (17.6) | 150 (17.6) | 69 (24.6) | 13 (21.3) | 834 (18.1) |
| 5 (deprived) | 431 (12.6) | 117 (13.7) | 33 (11.7) | 11 (18.0) | 592 (12.8) |
| Missing | 3 (0.1) | 0 (0.0) | 0 (0.0) | 0 (0.0) | 3 (0.1) |
| Smoking status | |||||
| Non-smoker | 1795 (52.5) | 492 (57.8) | 137 (48.8) | 37 (60.7) | 2461 (53.3) |
| Current smoker | 996 (29.1) | 237 (27.8) | 86 (30.6) | 16 (26.2) | 1335 (28.9) |
| Ex-smoker | 295 (8.6) | 67 (7.9) | 25 (8.9) | 5 (8.2) | 392 (8.5) |
| Missing | 334 (9.8) | 55 (6.5) | 33 (11.7) | 3 (4.9) | 426 (9.2) |
| Alcohol consumption? | |||||
| No | 312 (9.1) | 91 (10.7) | 21 (7.5) | 8 (13.1) | 432 (9.4) |
| Yes | 1628 (47.6) | 405 (47.6) | 122 (43.4) | 17 (27.9) | 2172 (47.1) |
| Missing | 1480 (43.3) | 355 (41.7) | 138 (49.1) | 36 (59.0) | 2009 (43.6) |
| CCI score | |||||
| 0 | 3153 (92.2) | 792 (93.1) | 270 (96.1) | 60 (98.4) | 4275 (92.7) |
| 1 | 162 (4.7) | 37 (4.3) | 6 (2.1) | 0 (0.0) | 205 (4.4) |
| 2 | 65 (1.9) | 15 (1.8) | 4 (1.4) | 0 (0.0) | 84 (1.8) |
| ≥ 3 | 40 (1.2) | 7 (0.8) | 1 (0.4) | 1 (1.6) | 49 (1.1) |
| Region | |||||
| East Midlands | 112 (3.3) | 30 (3.5) | 8 (2.8) | 2 (3.3) | 152 (3.3) |
| East of England | 424 (12.4) | 98 (11.5) | 22 (7.8) | 5 (8.2) | 549 (11.9) |
| London | 418 (12.2) | 102 (12.0) | 34 (12.1) | 6 (9.8) | 560 (12.1) |
| North East | 78 (2.3) | 26 (3.1) | 6 (2.1) | 3 (4.9) | 113 (2.4) |
| North West | 540 (15.8) | 129 (15.2) | 60 (21.4) | 14 (23.0) | 743 (16.1) |
| South Central | 496 (14.5) | 132 (15.5) | 39 (13.9) | 2 (3.3) | 669 (14.5) |
| South East Coast | 418 (12.2) | 84 (9.9) | 26 (9.3) | 9 (14.8) | 537 (11.6) |
| South West | 414 (12.1) | 110 (12.9) | 46 (16.4) | 10 (16.4) | 580 (12.6) |
| West Midlands | 378 (11.1) | 110 (12.9) | 30 (10.7) | 7 (11.5) | 525 (11.4) |
| Yorkshire and the Humber | 142 (4.2) | 30 (3.5) | 10 (3.6) | 3 (4.9) | 185 (4.0) |
| Epilepsy? | |||||
| No | 3328 (97.3) | 809 (95.1) | 267 (95.0) | 53 (86.9) | 4457 (96.6) |
| Yes | 92 (2.7) | 42 (4.9) | 14 (5.0) | 8 (13.1) | 156 (3.4) |
| Prescribed painkillers 3 months after the index date? | |||||
| No | 3241 (94.8) | 795 (93.4) | 264 (94.0) | 56 (91.8) | 4356 (94.4) |
| Yes | 179 (5.2) | 56 (6.6) | 17 (6.0) | 5 (8.2) | 257 (5.6) |
| Prescribed painkillers 1 month after the index date? | |||||
| No | – | – | 248 (88.3) | 51 (83.6) | – |
| Yes | – | – | 33 (11.7) | 10 (16.4) | – |
| First-time TASD | |||||
| 14G5: H/O dislocated shouldera | 393 (11.5) | 84 (9.9) | 33 (11.7) | 5 (8.2) | 515 |
| 7K6G300: closed reduction of dislocation of shoulder | 58 (1.7) | 24 (2.8) | 3 (1.1) | 2 (3.3) | 87 |
| N083100: recurrent joint dislocation, of shoulder region | 29 (0.8) | 8 (0.9) | 5 (1.8) | 1 (1.6) | 43 |
| N083A00: recurrent dislocation of shoulder – anterior | 113 (3.3) | 25 (2.9) | 25 (8.9) | 4 (6.6) | 167 |
| N083C00: recurrent subluxation of shoulder – anterior | 52 (1.5) | 14 (1.6) | 7 (2.5) | 1 (1.6) | 74 |
| S41..00: dislocation or subluxation of shoulder | 2081 (60.8) | 496 (58.3) | 147 (52.3) | 36 (53.5) | 2760 |
| S410.00: closed traumatic dislocation of shoulder | 75 (2.2) | 31 (3.6) | 4 (1.4) | 2 (3.3) | 112 |
| S410000: closed traumatic dislocation of shoulder joint, unspecified | 21 (0.6) | 7 (0.8) | 1 (0.4) | 0 (0.0) | 29 |
| S410100: closed traumatic dislocation of shoulder joint, anterior (subcoracoid) | 13 (0.4) | 2 (0.2) | 2 (0.7) | 0 (0.0) | 17 |
| S410111: anterior dislocation of shoulder | 87 (2.5) | 37 (4.3) | 11 (3.9) | 2 (3.3) | 137 |
| S412.00: closed traumatic subluxation, shoulder | 18 (0.5) | 2 (0.2) | 1 (0.4) | 0 (0.0) | 21 |
| S41z.00: dislocation of shoulder NOSa | 480 (14.0) | 121 (14.2) | 42 (14.9) | 8 (13.1) | 651 |
Prediction model: surgical cohort
Eight predictors were included in the model to predict re-dislocation in the surgery cohort (Table 24). With an effective sample size of 61 re-dislocation events, this yields an events-per-variable (EPV) number of 5.1 (61 events/12 regression coefficients), which is much smaller than the widely recommended EPV number of 10, indicating the likelihood of overfitting because of a small sample size.
| Predictor | Coefficient (SE) | HR (95% CI) | p-value |
|---|---|---|---|
| Age (years) | –0.1029 (0.0366) | 0.90 (0.84 to 0.97) | 0.0050 |
| Sex | 0.10275 (0.3825) | 1.11 (0.52 to 2.34) | 0.7887 |
| BMI (kg/m2) | –0.0777 (0.0606) | 0.93 (0.82 to 1.04) | 0.2015 |
| Smoking status (reference: non-smoker) | |||
| Smoker | –0.4653 (0.3444) | 0.63 (0.32 to 1.23) | 0.1767 |
| Ex-smoker | –0.0989 (0.5322) | 0.91 (0.32 to 2.57) | 0.8526 |
| Alcohol consumption (reference: non-drinker) | –0.3965 (0.4968) | 0.67 (0.25 to 1.79) | 0.4255 |
| Epilepsy | 0.9216 (0.4434) | 2.51 (1.05 to 5.99) | 0.0377 |
| Painkillers within 1 month of index date | 0.5171 (0.3848) | 1.68 (0.79 to 3.57) | 0.1791 |
| IMD 2004 [reference: 1 (affluent)] | |||
| 2 | 0.7099 (0.5146) | 2.03 (0.74 to 5.58) | 0.1677 |
| 3 | 1.0641 (0.5029) | 2.90 (1.08 to 7.77) | 0.0344 |
| 4 | 0.6600 (0.5127) | 1.93 (0.71 to 5.29) | 0.1980 |
| 5 (deprived) | 1.0757 (0.5667) | 2.93 (0.97 to 8.90) | 0.0577 |
Age and epilepsy were the only statistically significant predictors (at the p < 0.05 level). The apparent predictive performance of the model, as measured by the c-index, was moderate, with a c-index of 0.72 (95% CI 0.65 to 0.80), which dropped slightly to 0.67 after correcting for optimism (because of overfitting).
Prediction model: non-surgical cohort
Nine predictors were included in the model to predict re-dislocation in the non-surgery cohort (Table 25). With an effective sample size of 851 re-dislocation events, this yields an EPV number of 56.7 (851 events/15 regression coefficients), which is higher than the widely recommended EPV number of 10, indicating a sufficient sample size for model development and the minimal likelihood of overfitting.
| Predictor | Coefficient (SE) | HR (95% CI) | p-value |
|---|---|---|---|
| Age (years) | –0.0379 (0.0069) | 0.96 (0.95 to 0.98) | < 0.0001 |
| Sex | –0.2179 (0.0959) | 0.80 (0.67 to 0.97) | 0.0230 |
| BMI (kg/m2) | –0.0122 (0.0094) | 0.99 (0.97 to 1.01) | 0.1964 |
| Smoking status (reference: non-smoker) | |||
| Smoker | –0.0775 (0.0828) | 0.93 (0.79 to 1.09) | 0.3496 |
| Ex-smoker | –0.0039 (0.1340) | 1.00 (0.77 to 1.21) | 0.9770 |
| Alcohol consumption (reference: non-drinker) | –0.0336 (0.1142) | 0.97 (0.77 to 1.21) | 0.7687 |
| Epilepsy | 0.6377 (0.1598) | 1.89 (1.38 to 2.59) | 0.0001 |
| Painkillers within 1 month of index date | –0.0726 (0.1139) | 0.93 (0.74 to 1.16) | 0.5242 |
| CCI score (reference: 0) | |||
| 1 | –0.1202 (0.1687) | 0.89 (0.64 to 1.23) | 0.4763 |
| 2 | –0.0464 (0.2616) | 0.95 (0.57 to 1.59) | 0.8593 |
| 3 | –0.2470 (0.3804) | 0.78 (0.37 to 1.65) | 0.5161 |
| IMD 2004 [reference: 1 (affluent)] | |||
| 2 | 0.1178 (0.0975) | 1.12 (0.93 to 1.36) | 0.2271 |
| 3 | 0.0031 (0.1046) | 1.00 (0.82 to 1.23) | 0.9766 |
| 4 | 0.0770 (0.1065) | 1.08 (0.88 to 1.33) | 0.4697 |
| 5 (deprived) | 0.1619 (0.1169) | 1.18 (0.93 to 1.48) | 0.1662 |
Age, sex and epilepsy were statistically significant predictors (at the p < 0.05 level). The apparent predictive performance of the model, as measured by the c-index, was low (c-index 0.58, 95% CI 0.56 to 0.60), which dropped to 0.56 after correcting for optimism (because of overfitting).
Discussion
Main findings
The use of primary care data to predict the risk of re-dislocation following a first-time TASD is limited. However, the performances of the two models were markedly different, with better performance in the surgery cohort. Both models identified age and epilepsy as statistically significant predictors of re-dislocation, but in the non-surgery cohort sex was also identified as a statistically significant predictor.
The model developed in the surgery cohort showed moderate performance, with a c-index of 0.67, suggesting that the information collected has some predictive capacity but that additional information (risk factors) is needed to allow better predictions of re-dislocation. For the model developed in the non-surgery cohort, the predictive ability of the model was poor, with a c-index of only 0.57, and, therefore, has no use for the risk factors available in predicting re-dislocation in this cohort of patients.
Strengths and limitations
This study has several strengths. The models were developed using data routinely collected in electronic health-care records in primary care. Widely recommended statistical methodology was followed to develop and evaluate the models, including the exploration of complex relationships (e.g. non-linearity) in continuous measurements (e.g. age and BMI). Bootstrapping techniques were used to internally validate the models.
There are also some limitations. We were only able to include a small number of predictors, fewer than half of those identified in the clinical consensus. This had a considerable impact on our ability to develop models to accurately predict the risk of re-dislocation. There was also a considerable number of missing data (e.g. BMI, alcohol consumption). However, we followed recommended guidance and increased the number of imputations to 100 to account for the large number of missing data and any uncertainty in the imputation.24 Despite using a large electronic health records database (CPRD linked to HES), the number of eligible individuals, notably in the surgical cohort, was surprisingly small, with only 61 re-dislocation events during the study period. A small sample size can lead to overfitting. However, to counter the risk of overfitting, internal validation was carried out using bootstrapping to obtain unbiased estimates of model performance. Another limitation was the lack of a separate data set to carry out external validation, particularly to evaluate the model in the surgery cohort, which showed some predictive accuracy in the internal validation.
Conclusion
An insufficient number of data are routinely collected in CPRD and HES to allow for any reliable prediction modelling for shoulder re-dislocations after a first-time TASD in 16- to 35-year-olds.