A propensity score is the modelled probability of receiving treatment given the covariates, and the only thing it is for is making the treated and control groups comparable on those covariates. It is judged by whether the groups end up balanced, not by whether the model predicts treatment well. A model that predicts treatment perfectly has shown that the two groups do not overlap, which means there is no comparison left to make.
A high area under the curve is a warning, not a score
Every instinct trained on prediction problems says a model with an area under the curve of 0.97 is excellent and one at 0.53 is worthless. For a propensity model that reading is exactly backwards, and the third preset shows why. The model reaches 0.9653. It can look at a unit’s prior earnings and schooling and say, almost without error, whether that unit enrolled.
Which means the treated and control groups occupy different regions of the covariate space. Fifty of the ninety units have no counterpart on the other side at all, and the fitted scores reach exactly 1.0000 and 0.0000 — units that were certain to be treated, for whom no control exists anywhere in the data. The residual imbalance after weighting is 0.8804, against 0.1201 on the first preset and 0.0005 on the second.
Watch the sign on the third preset, because it is the detail that gives the game away. Prior earnings start 1.5540 standard deviations below in the treated group and finish 0.8804 above. The weighting did not fail to close the gap. It reversed it, because a handful of units with extreme scores received enormous weights and dragged the weighted mean past the target.
This is not an artefact of three hand-picked datasets. Simulating 300 paired draws — one weakly confounded, one strongly confounded, same sample size, same covariates — the higher-discrimination model left the worse balance in 300 out of 300. Mean residual imbalance was 0.50260 at a mean area under the curve of 0.9328, against 0.00504 at 0.5695. That is a factor of 99.7, and 22.7 times as many units stranded off common support.
What the balance table is actually measuring
A standardised mean difference is the gap between the group means divided by the pooled standard deviation. Dividing by the spread is what makes it portable: earnings in thousands and schooling in years produce numbers on the same scale, so one table can be read across covariates measured in entirely different units.
The 0.1 threshold is a convention, not a test. Nothing changes at 0.0999 that does not change at 0.1001, and the page reports the number rather than a verdict for that reason. On the first preset schooling lands at 0.0890 and prior earnings at 0.1201 — one inside, one outside, and the honest description is that the design supports a weighted comparison on one covariate and a qualified one on the other.
There is no p-value here on purpose. Testing whether a covariate difference is significant confuses the sample with the population: with enough rows a trivial imbalance becomes significant, and with few rows a serious one does not. The size of the imbalance is what biases the estimate, so the size is what gets reported.
Units with no counterpart
The overlap table gives the range of fitted scores in each group and counts how many units fall outside the other group’s range entirely. Those units are not difficult to match. They are impossible to match, and no weighting scheme invents a counterpart that the data does not contain.
Two units on the second preset fall off support, eighteen on the first, fifty on the third. An estimate that covers the third preset’s full sample is largely an extrapolation from the functional form of the model rather than a comparison between similar units, and the usual remedies — trimming the tails, restricting to the region of common support — change what population the answer applies to. That is a real cost, not a technicality: the question quietly becomes a different question.
The check that the fit converged
Every run prints the mean fitted score beside the treated fraction. They match to the last decimal, and they must: a logistic model with an intercept satisfies the score equation that the sum of observed minus fitted values is zero, so the average fitted probability equals the observed proportion exactly. Across the suite that identity holds to 3.3e-16.
It proves the optimiser landed where it should and nothing more. A converged model can still be the wrong model — the wrong covariates, the wrong functional form, a missing interaction — and the identity is blind to all of it. The area-under-the-curve figure is checked the same way, against a brute-force count of every treated-control pair, so the number in the tile is not resting on one implementation of one formula.
What balance does not buy you
A clean balance table says the groups match on the covariates in the table. It says nothing at all about the ones that were never measured, and in most observational settings those are the ones doing the damage. Motivation, health at baseline, whatever made someone seek the programme out — if it is not a column it is not balanced, and the propensity score cannot know it is missing.
That is why balance is a precondition rather than a conclusion. Once the table is clean the remaining question is how strong an unmeasured confounder would have to be to account for the result, which is what an E-value puts a number on. The weights themselves carry a separate cost in precision that the weighting page measures directly, and designs that sidestep the problem altogether — an instrument, a policy change, a cutoff — buy their credibility with assumptions of their own.
Reporting it
Report the balance table, not the model. Readers need the standardised mean difference for every covariate before and after adjustment, the number of units off common support, and the sample size that survives any trimming. The coefficients of the propensity model itself are of no interest to anyone — they are a means to a weighting scheme, and interpreting them as a model of who enrols is a different exercise with different requirements.
If the area under the curve is reported at all, report it as a diagnostic of overlap with the off-support count beside it. Presenting it as evidence the adjustment worked is the single most common error in this literature, and on the third preset it would have let a design with fifty stranded units and a reversed imbalance be described as a model that fits well.
Sources and methodology
References for the propensity score and for judging it by balance.
Method. The propensity model is fitted by iteratively reweighted least squares and checked against the score identity that the mean fitted probability equals the treated fraction, which holds to 3.3e-16. The area under the curve is computed from the rank-sum identity and cross-checked against a brute-force count over every treated-control pair. The discrimination-versus-balance result comes from 300 paired simulated draws. That engine is verified on every change against 556 assertions. The count and the per-case breakdown are published on the formula verification page.