Scott OatleySociology / UCL
← Guides

Handling missing data: an interactive guide

Mechanisms, methods and evidence, with simulations you can run in your browser

This page goes with the SGSSS Winter School session Handling missing data (10 December 2026). It covers the same ground as the slides, in order: what causes missing data, what you can and can't test, how to judge a method, and how complete records analysis, FIML and multiple imputation behave in practice.

Every simulation on this page runs live in your browser. It uses the design from Oatley, Gayle and Connelly (forthcoming), so you can check the claims for yourself rather than taking them on trust. Every dataset is generated from the same model:

y = 500 + 250 x1 + 400 x2 + 375 x3 + ε

The three covariates are correlated (r between 0.30 and 0.40), R² is 0.65, and only x2 has missing values. Because we made the data, we know the true coefficients. That is what lets us see whether a method gets them right.

Slides, Stata .do files, R scripts and datasets for every part are on the training page; direct links are in section 13.

1Missing data and its mechanisms

Missing values on their own cost you precision. Whether they also bias your results depends on why the values are missing. Rubin (1976) distinguishes three mechanisms:

MechanismThe chance a value is missing depends onExample (an income question)
MCAR (missing completely at random)Nothing: it is unrelated to any dataA server fault loses a random batch of responses
MAR (missing at random)Only things you have observedOlder people skip the question, and age is recorded
MNAR (missing not at random)The missing values themselves, beyond anything observedHigh earners withhold income because it is high

There is an important split inside MAR. If a covariate in your model drives the missingness, dropping incomplete cases selects on that covariate. The relationship between y and the x's is left intact, so a complete records analysis is still unbiased. If the outcome drives the missingness, dropping cases selects on y and distorts the relationship you are trying to estimate. Both are “MAR”, but they have opposite consequences.

Explore a mechanism
x2 observedx2 missing (we can see it because we simulated it)
Things to try
  • With driven by x1, the missing points pile up at high x1 in the left panel. The complete records estimate of the x2 coefficient still lands close to 400.
  • Switch to driven by y. The missing points now sit at high y in the right panel, and the complete records estimate drops below 400.
  • Switch to MNAR. Neither panel shows a clear pattern, because the missingness depends on a part of x2 that nothing else predicts. Yet the estimate is badly off.

2What the data can and can’t tell you

The standard diagnostics compare people with and without a missing value on the variables you can see: t-tests, chi-squared tests, a logistic regression of a missingness indicator, or Little’s MCAR test (mcartest in Stata, mcar_test() in R). If the groups differ, the data are not MCAR.

What no test can do is compare them on the values that are missing. The table below uses the sample from section 1. The last row is only possible because this is a simulation.

Comparing cases with x2 observed and x2 missing (current sample)
Things to try
  • Under MCAR nothing differs, and that's correct.
  • Under either MAR mechanism the observed variables differ, so you can reject MCAR.
  • Under MNAR the observed variables barely differ, so the diagnostics look reassuring, but the true x2 values of the missing cases are very different. This is why MAR versus MNAR has to be argued from what you know about the data, not tested.

How much missing data is too much? Rules of thumb (5%, 10%) don’t tell you whether results are biased. The mechanism decides whether a method is valid; the proportion missing decides how much precision you lose. You can check this in the lab in section 5: complete records analysis is unbiased at 90% missing when x1 drives the missingness, and biased at 50% when y does.

3Three ways to judge a method

“Does this method work?” is really three separate questions. Methods fail them in every combination, so keep them apart.

CriterionQuestionMeasured by
AccuracyOver many samples, is the average estimate equal to the true value?Absolute percentage bias
EfficiencyAmong accurate methods, how much do the estimates vary from sample to sample?Empirical standard error
CalibrationDo the standard errors the method reports match how much its estimates actually vary?Relative error in the SE; coverage of 95% intervals

Bias matters most because it doesn’t shrink as the sample grows: a bigger sample just homes in on the wrong value with more confidence. Efficiency has a useful interpretation: if one method’s sampling variance is half another’s, it gets as much out of the data as the other would from twice as many cases.

A single dataset can’t show you any of this, because you only get one estimate. Section 4 shows what one sample looks like; section 5 repeats the experiment hundreds of times so you can see all three criteria.

4Nine methods on one sample

These are the nine methods compared in the paper, grouped by when they are valid:

GroupMethodsValid when
InadequateMean imputation; regression imputation (filling in predicted values with no added noise); missing-indicator (dummy) methodAlmost never in real data
Conditionally validComplete records analysis (listwise deletion); inverse probability weighting (IPW)Missingness unrelated to y given x (deletion); correct model for who is missing (IPW)
Gold standardFull information maximum likelihood (FIML); multiple imputation (MI), with or without auxiliary variablesData are MAR
Estimates and 95% intervals from every method, using the sample from section 1
Change the mechanism or draw a new sample in section 1, then refit.

Dashed lines mark the true values. FIML standard errors here come from 200 bootstrap samples; MI uses m = 20.

Things to try
  • Look at the x1 panel. x1 has no missing values, yet mean imputation and the missing-indicator method move its coefficient a long way. Filling in one variable distorts its correlations with the others.
  • Compare the width of the complete records interval with FIML and MI. Under covariate-driven MAR they all centre in the same place, but FIML and MI are narrower.
  • One sample can mislead. Draw a few new samples and refit, then go to section 5.

5Repeated-sampling lab

This is the simulation study in miniature. Pick a mechanism and a missingness rate, then generate hundreds of datasets, apply every method to each, and see how the estimates behave across samples. The table reports the three criteria; the plots show the full spread of estimates.

Press Run. 200 replications take a few seconds.

Bias, SE error and coverage are red when they go past roughly what simulation noise allows (bias over 2%, SE error beyond ±10%, coverage below 0.92). FIML's bootstrap standard errors are too slow for the lab, so its SE error and coverage are left blank. In the paper they were well calibrated.

Things to try
  • Driven by x1, 50%: every principled method is unbiased with coverage near 0.95. Ad hoc methods are biased, and regression imputation’s intervals almost never contain the truth.
  • Driven by y, 50%: complete records analysis becomes biased with coverage around 0.6, while its SE error stays near zero. Nothing in its own output warns you.
  • MNAR: every method fails. Auxiliary variables reduce the bias but can't remove it.
  • Switch the coefficient to x1 to see how far the damage from ad hoc methods spreads.

6Why mean imputation fails

Mean imputation sets every missing x2 to the same value. With a single predictor the attenuation happens to cancel out, so it can look harmless. With correlated covariates, every imputed case sits on a flat line in the x2–x1 relationship. The regression then shares out the explained variance wrongly among the covariates, and every coefficient is biased, including those on fully observed variables.

The only setting where mean imputation is harmless for regression coefficients is when the covariates are uncorrelated, which real data almost never are. Simulation studies that use independent covariates therefore understate its damage.

Bias as the correlation between x1 and x2 increases (50% missing, driven by x1)
Mean imputationComplete records analysis

Only the x1–x2 correlation changes here. x2 is still correlated with x3 (r = 0.35), so mean imputation stays somewhat biased even at zero. It is only harmless when x2 is uncorrelated with every other covariate. Complete records analysis stays unbiased throughout; any wobble in its line is simulation noise, which is largest at high correlations.

7Full information maximum likelihood

FIML neither deletes nor fills in anything. Each case contributes to the likelihood whatever it has observed. A complete case contributes the joint density of y, x1, x2 and x3. A case missing x2 contributes the joint density of y, x1 and x3, with x2 integrated out. Because the means and covariances are shared across both kinds of case, the partial records still inform every parameter.

FIML assumes MAR and multivariate normality. Two practical points matter:

* Stata
sem (y <- x1 x2 x3), method(mlmv)

# R
fit <- lavaan::sem('y ~ x1 + x2 + x3', data = dat, missing = "fiml", fixed.x = FALSE)

8Inside multiple imputation

Multiple imputation repairs the two flaws of regression imputation. Each missing value gets a predicted value plus random noise, drawn using regression parameters that are themselves uncertain. This is repeated m times. You analyse each completed dataset and combine the results with Rubin’s rules:

QuantityFormulaMeaning
Pooled estimateQ̄ = (1/m) Σ Q̂jThe average across imputations
Within-imputation varianceW̄ = (1/m) Σ ŴjOrdinary sampling variance
Between-imputation varianceB = Σ (Q̂j − Q̄)² / (m − 1)Uncertainty due to the missing data
Total varianceT = W̄ + (1 + 1/m) BWhat the pooled standard error is based on
Imputing the sample from section 1
imputed valuestrue value (known only because this is simulated)
Things to try
  • In the left panel each row is one missing case. The imputations scatter around the truth rather than sitting on it, and that scatter is what carries the uncertainty.
  • Tick Use the auxiliary variables. The imputations tighten around the true values, B shrinks, and FMI (the fraction of missing information) falls.
  • Raise m. The pooled estimate stabilises, but FMI and the standard error barely change. More imputations reduce simulation noise; they don't add information.

Build the imputation model carefully:

9Auxiliary variables and the number of imputations

An auxiliary variable is one that isn’t in your analysis model but predicts the missing values, or whether they are missing. Earlier waves, linked administrative data and related survey items are typical sources. An auxiliary only helps where your analysis variable is missing, so it must be observed in those cases.

How much precision do auxiliaries and extra imputations buy? (driven by x1)

Effective sample size relative to complete records analysis = (empirical SE of deletion / empirical SE of the method)², for the x2 coefficient. With a few hundred replications expect ±0.2 of simulation noise.

In the paper (1,000 replications, 90% missing), auxiliaries raised MI’s effective sample size from 1.48 to 2.09 times that of deletion. Going from 10 to 100 imputations added only a little more, at about nine times the run time. Common guidance on m: at least the percentage of incomplete cases (White, Royston and Wood, 2011), or von Hippel’s (2020) two-stage rule based on FMI. Below about 50% missing, m = 10 and m = 100 were indistinguishable.

10Published results

These are the full-scale results from the paper (1,000 replications, n = 1,000). Your runs in the lab should land close to them.

Covariate-driven MAR, 50% missing: coefficient on x2

Precision as missingness rises

Outcome-driven MAR and MNAR, 50% missing

Source: Oatley, Gayle and Connelly (forthcoming), Tables 4–6.

11A decision guide

Given the covariates in your model, could the chance of a value being missing be related to the outcome?
Could it depend on the missing values themselves, in ways nothing you observe captures?
Answer both questions.

Whatever the answers, don’t use mean, mode or median imputation, regression imputation or missing-indicator methods to estimate effects.

12Check yourself

Which mechanism is this?

A fieldwork tablet crashes on random days and loses those interviews.

MCAR. Deletion is unbiased; you only lose precision.

Older respondents skip a wellbeing item more often. Age is in the model, and wellbeing is a covariate.

MAR, covariate-driven. Deletion is unbiased. FIML or MI would be more precise.

In a model of GCSE attainment, parental education is missing more often for pupils with low attainment.

MAR, but outcome-driven. Deletion is biased. Use FIML, MI or IPW, with attainment in the imputation model.

People with high debts are less likely to report them, and nothing you observe predicts this.

MNAR. No standard method is guaranteed. Use auxiliaries, sensitivity analysis, and say so in the write-up.

Which method would you use?

40% missing on parental income; missingness is predicted by region and school type, both in your model.

Complete records analysis is valid. MI or FIML with auxiliaries would be more precise.

25% missing on a wellbeing covariate; pupils with poor attendance (your outcome) skip it more.

FIML or MI, with attendance in the imputation model.

Self-reported drinking is missing more often for heavy drinkers, and nothing observed predicts it well.

MI with auxiliaries, plus sensitivity analysis and an honest write-up.

13Code and reading

Session files

SessionStataR
1. Diagnosing missing data (sections 1–2)session one.dosession one.R
2. Deletion, single imputation and missing indicators (sections 4–6)session two.dosession two.R
3. FIML (section 7)session three.dosession three.R
4. Multiple imputation (sections 8–9)session four.dosession four.R

Datasets for session 1: datasetone.dta, datasettwo.dta, datasetthree.dta.

Multiple imputation in Stata

mi set mlong
mi register imputed age gender education income
mi register regular aux1 aux2
mi impute chained (regress) age income ///
                  (logit)   gender     ///
                  (ologit)  education  ///
                  = aux1 aux2, add(20) rseed(12345)
mi estimate: regress income age i.gender i.education
mi estimate, vartable dftable      // RVI, FMI

Multiple imputation in R

library(mice)
meth <- make.method(df)
meth[c("age", "income")] <- "pmm"
meth["gender"]    <- "logreg"
meth["education"] <- "polr"
meth[c("aux1", "aux2")] <- ""          # predictors only
pred <- quickpred(df, minpuc = 0.2)    # keep the outcome in
imp  <- mice(df, m = 20, method = meth, predictorMatrix = pred, seed = 123)
summary(pool(with(imp, lm(income ~ age + gender + education))))

What to report

Reading

Carpenter, J.R. and Kenward, M.G. (2012) Multiple Imputation and its Application. Wiley.

Collins, L.M., Schafer, J.L. and Kam, C.-M. (2001) A comparison of inclusive and restrictive strategies in modern missing data procedures. Psychological Methods, 6(4), 330–351.

Enders, C.K. (2010) Applied Missing Data Analysis. Guilford.

Little, R.J.A. (1992) Regression with missing X’s: a review. Journal of the American Statistical Association, 87(420), 1227–1237.

Little, R.J.A. and Rubin, D.B. (2019) Statistical Analysis with Missing Data, 3rd edn. Wiley.

Madley-Dowd, P. et al. (2019) The proportion of missing data should not be used to guide decisions on multiple imputation. Journal of Clinical Epidemiology, 110, 63–73.

Morris, T.P., White, I.R. and Crowther, M.J. (2019) Using simulation studies to evaluate statistical methods. Statistics in Medicine, 38(11), 2074–2102.

Oatley, S., Gayle, V. and Connelly, R. (forthcoming) A simulation study comparing missing data handling strategies: accuracy, efficiency and calibration under MCAR, MAR and MNAR.

Rubin, D.B. (1976) Inference and missing data. Biometrika, 63(3), 581–592.

Seaman, S.R. and White, I.R. (2013) Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 22(3), 278–295.

von Hippel, P.T. (2009) How to impute interactions, squares, and other transformed variables. Sociological Methodology, 39(1), 265–291.

White, I.R. and Carlin, J.B. (2010) Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Statistics in Medicine, 29(28), 2920–2931.

White, I.R., Royston, P. and Wood, A.M. (2011) Multiple imputation using chained equations: issues and guidance for practice. Statistics in Medicine, 30(4), 377–399.