Prediction models and overfitting

What a clinical prediction model is, the steps to develop one, why its apparent performance is optimistic, and how large the development data must be.

Explaining and predicting

Regression is used for two different purposes. In an aetiological study the question is the effect of one exposure, and the other variables are there to remove confounding: the coefficient of the exposure is the result. In a prediction study no coefficient is the result: the result is the predicted risk of each patient, and the question is whether these risks are accurate (calibration) and separate patients with and without the outcome (discrimination) in patients who were not used to build the model (Steyerberg 2019). A predictor need not be causal; it must only carry information that is available at the moment of prediction.

A diagnostic model predicts the presence of a disease now (e.g. pulmonary embolism in a patient with chest pain); a prognostic model predicts a future event (e.g. a cardiovascular event within 10 years). The tools are the same.

The steps

  1. Define the outcome and the moment of prediction. Who are the patients, when is the prediction made, and what is predicted by when (a time horizon for prognostic models).
  2. Choose candidate predictors from knowledge, not from the data: what is measured routinely, reliably and before the outcome.
  3. Check the sample size for the number of candidate parameters (below).
  4. Handle missing values (exploration, imputation; see the example).
  5. Specify the model: non-linear effects of continuous predictors (splines), interactions only when expected, no dichotomising of continuous variables.
  6. Estimate it, possibly with selection or penalisation (shrinkage).
  7. Evaluate the performance — discrimination, calibration, overall accuracy and clinical usefulness — and correct it for optimism by internal validation.
  8. Validate it in new patients (another period, centre or population) and, if needed, update it.
  9. Present it so that others can use it (the full equation, a calculator) and report it following TRIPOD+AI (Collins et al. 2024).

clinpred covers steps 3 to 9.

Overfitting and optimism

A model fitted to a data set describes that data set best, noise included. Its apparent performance, measured in the same patients, is therefore optimistic: in new patients the discrimination will be lower and the predictions too extreme (high risks too high, low risks too low). The more parameters relative to the number of events, the larger the optimism.

The simulation below shows it. Each data set has 300 patients (about 45 events) and three real predictors; noise predictors are then added. The apparent C-statistic (AUC) keeps rising with every useless predictor, while the AUC in a large independent sample falls. The bootstrap optimism-corrected AUC, computed by clinpred’s own validation engine, removes most of the optimism. It removes less of it when the parameters approach the number of events: there the data are too few even to estimate the optimism, which is one more reason to respect the sample size (below).

Figure 1: Apparent, bootstrap-corrected and true (new-patient) AUC of logistic models with 3 real predictors and an increasing number of noise predictors; 300 patients, means of 12 simulated data sets.

With 30 noise predictors (33 parameters for about 45 events) the apparent AUC is 0.864, but in new patients it is 0.664; the bootstrap says 0.746. The bootstrap also estimates how much the predictions are too extreme: the calibration slope of the model in new patients, 0.43 here (1 would be perfect). This is the uniform shrinkage factor: multiplying the coefficients by it would make the predictions about right. The validation page explains how the bootstrap does this.

The remedies are, in order: fewer candidate parameters (pre-specification from knowledge), more data, and penalised estimation (shrinkage, elastic net). Selection by p-values or AIC does not remove the optimism — it adds its own — unless it is repeated inside the validation, as clinpred does.

How many patients?

The old rule of 10 events per variable is too crude. Riley et al. (2020) (and Richard D. Riley et al. (2019a), Richard D. Riley et al. (2019b) for continuous and time-to-event outcomes) derive the minimum sample size from explicit targets:

  1. expected shrinkage of at least 0.9 (optimism in the predictions of at most 10%);
  2. optimism in the R² (Nagelkerke) of at most 0.05;
  3. precise overall risk (the intercept within ±0.05), or, for a continuous outcome, a precise residual SD and mean.

The criteria depend on the number of candidate parameters (every spline term, every factor level, every candidate interaction counts, whether or not it is kept), the outcome proportion, and the anticipated Cox–Snell R² of the model. clinpred’s Sample size adequacy table computes the three criteria from the anticipated R² you enter or from the model’s own optimism-corrected R², and compares them with the data you have (see the logistic example).

Back to top

References

Collins, Gary S, Karel G M Moons, Paula Dhiman, et al. 2024. “TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods.” BMJ 385 (April): e078378. https://doi.org/10.1136/bmj-2023-078378.
Riley, Richard D, Joie Ensor, Kym I E Snell, et al. 2020. “Calculating the Sample Size Required for Developing a Clinical Prediction Model.” BMJ, March, m441. https://doi.org/10.1136/bmj.m441.
Riley, Richard D., Kym I. E. Snell, Joie Ensor, et al. 2019a. “Minimum Sample Size for Developing a Multivariable Prediction Model: Part i – Continuous Outcomes.” Statistics in Medicine 38 (7): 1262–75. https://doi.org/10.1002/sim.7993.
Riley, Richard D, Kym IE Snell, Joie Ensor, et al. 2019b. “Minimum Sample Size for Developing a Multivariable Prediction Model: PART II - Binary and Time-to-Event Outcomes.” Statistics in Medicine 38 (7): 1276–96. https://doi.org/10.1002/sim.7992.
Steyerberg, Ewout W. 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. In Statistics for Biology and Health. Springer International Publishing. https://doi.org/10.1007/978-3-030-16399-0.