R syntax
r0 <- do.call(cpLogistic, c(args, list(missExplore = TRUE)))Missing values
Exploring missing values, and developing and validating the model with multiple imputation by chained equations instead of complete cases.
Question. 108 patients of the heart cohort lack cholesterol or body mass index, and the previous examples simply dropped them. Is that harmless? What changes if the missing values are imputed?
Why it matters
Dropping patients with a missing value (complete-case analysis) loses information and, worse, can bias the model if missingness is related to the outcome. A prediction model also needs a rule for patients with missing values when it is used. Multiple imputation by chained equations is the usual remedy (Steyerberg 2019): each missing value is filled in several times from a model of that variable given all the others including the outcome, the analysis is repeated in each completed data set, and the results are combined.
Step 1: explore
Under Missing values, tick Missing values (exploration).
Missing Values (Development Data)
| % missing | ||||||
|---|---|---|---|---|---|---|
| Variable | Role | Missing | % | Non-events | Events | p |
| event10 | Outcome | 0 | 0.0 % | |||
| sbp | Predictor | 0 | 0.0 % | |||
| chol | Predictor | 40 | 1.7 % | 1.7 % | 1.4 % | .611 |
| bmi | Predictor | 70 | 2.9 % | 3.1 % | 1.9 % | .206 |
| age | Predictor | 0 | 0.0 % | |||
| sex | Predictor | 0 | 0.0 % | |||
| smoking | Predictor | 0 | 0.0 % | |||
| diabetes | Predictor | 0 | 0.0 % | |||
| Note. Share of rows with the predictor missing among non-events and events (chi-square test): a difference means that missingness depends on the outcome (not missing completely at random). | ||||||
Missing Value Patterns
| Missing | Variables | Rows | % |
|---|---|---|---|
| none (complete) | 0 | 2292 | 95.5 % |
| bmi | 1 | 68 | 2.8 % |
| chol | 1 | 38 | 1.6 % |
| chol, bmi | 2 | 2 | 0.1 % |
| Note. The most frequent combinations of missing variables in the development rows (at most 10). | |||
Only two variables have missing values, 2.9% for bmi and 1.7% for cholesterol, almost never together. The share missing does not differ clearly between patients with and without an event (bmi: 3.1% vs 1.9%, p = 0.206). Missingness that depends on the outcome would be a warning against complete-case analysis. When the model is validated in a split or an external file, the table adds the share missing there.
Step 2: impute
Missing values: Multiple imputation, 5 imputations, 10 iterations. clinpred imputes numeric variables by predictive mean matching (an observed value of a similar patient), factors by logistic or multinomial regression, with the outcome in every imputation model (for Cox models, the event and its cumulative hazard (White and Royston 2009)).
R syntax
r1 <- do.call(cpLogistic, c(args, list(missMethod = "mi", miM = 5)))Data
| Data set | N | Events | % | Excluded (missing) |
|---|---|---|---|---|
| Development | 2400 | 369 | 15.4 % | 0 |
| Note. Multiple imputation: 5 imputations by chained equations (predictive mean matching, 10 iterations; the outcome in the imputation models) of 108 development rows with a missing predictor; validation data imputed separately. Coefficients pooled by Rubin's rules, performance averaged over the imputations. Rows with a missing outcome are excluded. | ||||
Odds Ratios
| Crude | Adjusted | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| OR | Lower | Upper | p | OR | Lower | Upper | p | ||
| sbp | per 10 units | 1.24 | 1.156 | 1.33 | < .001 | 1.20 | 1.110 | 1.29 | < .001 |
| chol | per unit | 1.22 | 1.098 | 1.36 | < .001 | 1.27 | 1.132 | 1.42 | < .001 |
| bmi | per unit | 1.03 | 1.007 | 1.06 | .012 | 1.02 | 0.993 | 1.05 | .160 |
| age | Q3 vs Q1 (66.25 vs 53) | 1.16 | 0.853 | 1.58 | < .001 | 2.07 | 1.234 | 3.46 | < .001 |
| sex | female | 1.00 | < .001 | 1.00 | < .001 | ||||
| male | 1.54 | 1.225 | 1.92 | < .001 | 1.78 | 1.283 | 2.46 | < .001 | |
| smoking | never | 1.00 | < .001 | 1.00 | < .001 | ||||
| former | 1.11 | 0.846 | 1.45 | .459 | 1.15 | 0.870 | 1.51 | .332 | |
| current | 2.16 | 1.648 | 2.84 | < .001 | 2.21 | 1.672 | 2.93 | < .001 | |
| diabetes | no | 1.00 | < .001 | 1.00 | < .001 | ||||
| yes | 2.30 | 1.640 | 3.23 | < .001 | 2.28 | 1.593 | 3.27 | < .001 | |
| Note. Crude: each predictor alone, on the same rows. Adjusted: in the prediction model. | |||||||||
| Note. Spline predictors: odds ratio between the third and the first quartile; the p is the likelihood ratio test of the whole (non-linear) association. | |||||||||
| Note. The p on a factor's reference row is the likelihood ratio test of the factor. | |||||||||
| Note. With interactions, an adjusted odds ratio is the effect at the reference level (factor) or median (covariate) of the variables it interacts with; see Effects Within Subgroups. | |||||||||
| Note. Crude and adjusted effects pooled over the imputations by Rubin's rules. | |||||||||
All 2400 patients are now used. The pooled odds ratios (Rubin’s rules) barely differ from the complete-case ones: with 4.5% of patients incomplete and missingness unrelated to the outcome, complete-case analysis was harmless here. It would not be with 30% missing.
Step 3: validate with imputed data
Performance in the Development Data
| 95% CI | ||||||
|---|---|---|---|---|---|---|
| Measure | Apparent | Lower | Upper | Optimism | Corrected | |
| Discrimination | C-statistic (AUC) | 0.68934 | 0.01098 | 0.67834 | ||
| Discrimination slope | 0.06713 | 0.00519 | 0.06195 | |||
| Calibration | Calibration-in-the-large | 0.00001 | -0.00079 | 0.00079 | ||
| Calibration slope | 0.99997 | 0.06871 | 0.93129 | |||
| O/E ratio | 1.00001 | -0.00206 | 1.00206 | |||
| ICI | 0.00437 | -0.00137 | 0.00575 | |||
| E50 | 0.00426 | -0.00105 | 0.00519 | |||
| E90 | 0.00788 | -0.00270 | 0.01058 | |||
| Emax | 0.02076 | 0.03724 | 0.00000 | |||
| Overall | Brier score | 0.12143 | -0.00146 | 0.12289 | ||
| Scaled Brier score | 0.06670 | 0.01121 | 0.05550 | |||
| Nagelkerke R² | 0.10594 | 0.01612 | 0.08984 | |||
| Cox–Snell R² | 0.06104 | 0.00929 | 0.05175 | |||
| Note. Bootstrap optimism correction (Harrell): 100 samples (100 usable); corrected = apparent − optimism. | ||||||
| Note. An error measure whose estimated optimism exceeds its apparent value (a noisy measure such as Emax) is shown corrected as 0. | ||||||
| Note. Averaged over the imputations (no confidence intervals); internal validation within each imputation. | ||||||
Performance is computed with the pooled model in each imputed data set and averaged; the bootstrap optimism is estimated within each imputation (Wahl et al. 2016), with the whole procedure — selection included — repeated in every resample. Validation data (a split or an external file) are imputed separately, so that no information from them reaches the model.
The other options
| Method | What it does | When |
|---|---|---|
| Complete cases | drops incomplete rows | few missing values, unrelated to the outcome |
| Missing category / indicator | a (missing) level for factors; for numeric variables, the development median plus an indicator predictor |
missingness that is itself informative and will also occur in practice (a test not ordered) |
| Single imputation | one chained-equations imputation | a quick look; standard errors too small |
| Multiple imputation | m imputations, pooled | the default choice for development |
With the missing indicator method the model includes the indicators and stores the development medians; Export model writes both, so the published model can be applied to new patients with missing values.
References
Steyerberg, Ewout W. 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. In Statistics for Biology and Health. Springer International Publishing. https://doi.org/10.1007/978-3-030-16399-0.
Wahl, Simone, Anne-Laure Boulesteix, Astrid Zierer, Barbara Thorand, and Mark A. van de Wiel. 2016. “Assessment of Predictive Performance in Incomplete Data by Combining Internal Validation and Multiple Imputation.” BMC Medical Research Methodology 16 (1). https://doi.org/10.1186/s12874-016-0239-7.
White, Ian R., and Patrick Royston. 2009. “Imputing Missing Covariate Values for the Cox Model.” Statistics in Medicine 28 (15): 1982–98. https://doi.org/10.1002/sim.3618.