Towards a standardized evaluation
of imputation methodology

Potential pitfalls in simulation studies and a proposed course of action

H.I. Oberman & G. Vink

Utrecht University

Missing data in the wild

Who of you has ever done some statistical consulting for empirical researchers?

And of you who has *never* come across a real-world dataset with missing values?

It’s always nice to be able to suggest something better than a biasing default such as list-wise deletion,

but *however advanced* the missing data method that you suggest may be, the issue remains:

missing values are by definition missing.

It is seldom possible to recover the unobserved true data that is obfuscated by the missingness.

So you may never be certain that the solution you suggested, was the right one.

Making up missing data

That is, unless you’re doing a simulation study.

In simulation studies, it is possible to start out with a complete dataset and impose missingness yourself.

Then, as the simulator, you’re able to determine exactly how well your suggested solution fixes the problem at hand,

within the confined space of the simulation design.

Lack of common ground

But that brings us the the *real* problem I’d like to discuss today:

simulators create their own missing data problems,

and they do not sufficiently report their simulation designs.

The lack of common ground in evaluating imputation methodology

may lead to suboptimal use in practice,

because simulation results are not comparable

and hard to translate into real-world/empirical missing data problems

Requiring reporting guidelines?

Then just require simulators to report on their designs?

The case of spurious missingness

MAR may be spurious.

All of them validly generated, but only two are valid missingness

Standardizing the evaluation

Standardizing the evaluation of the simulation studies!

With standardized evaluation, simulation studies may be

1) better documented

2) more comparable

3) easier to interpret/extrapolate from

Developing a checklist

And sharing (pseudo) code.

To aid fellow simulators, we are developing a checklist.

Take-aways

And hopefully, with this standardized evaluation,

you will have more confidence in the suggestions you do as missing data consultant!

References

Oberman, H. I., & Vink, G. (2023). Toward a standardized evaluation of imputation methodology. Biometrical Journal, e2200107.

Checklist on Research Equals.

Publication archive on Zenodo .

The checklist

Aims

e.g., simulation scope

• simulation design (incl. pseudocode or flow diagram)
• required level of precision (incl. number of simulation repetitions)

Data-generating mechanisms

e.g., data generation and missingness generation

• data source (incl. model-based or design-based, sampling variance)
• data characteristics (incl. multivariate relations and data structures such as clustering)
• missingness mechanisms (incl. type or functional form of the missing data model)
• missingness patterns (incl. missingness proportion)

Estimands

e.g., complete data target

Methods

e.g., missing data methods, analytic method and inference pooling rules

• imputation methods (incl. parameters such as the number of imputations)
• estimation method (incl. reference methods such as complete case analysis)
• methods used to construct standard errors and confidence intervals

Performance measures

e.g., evaluation of estimates and imputed values

• statistical properties (incl. comparative performance, if applicable)
• validity of imputations (incl. imputation model fit and distributional characteristics)