A hundred and fifty observations and four correlated predictors, cross-validated over a ridge path. The reference five-fold split picks a penalty of 0.01. Re-draw the folds twelve times on exactly the same rows and the choice comes back as 0.01, 0.05, 0.01, 0.01, 0.1, 0.1, 0.05, 0.05, 0.1, 0.05, 0.1, 0.05 — three distinct values across a tenfold range. The error those choices achieve varies by 2.00%. The penalty is not determined by the data anything like as tightly as a single reported value implies, and the reason it seldom matters is visible in the path: the error surface is flat right across that region.
Re-drawing the folds returns 3 different penalties, from 0.01 to 0.1
That is a 10.0-fold range on identical rows, and the error those choices achieve varies by only 2.00%. The penalty is far less determined by the data than a single reported value implies — and the reason it seldom matters is on the same screen, in a path whose error is nearly flat across that whole region.
Penalty chosen here
0.01
error 1.0280
Across twelve re-splits
0.01 to 0.1
3 distinct values
Error varies by
2.00%
far less than the choice does
Training error is lower by
2.2%
which is why held-out data is needed
Twelve fold assignments, one dataset
The penalty chosen and the error achieved under each fold assignment
Assignment
Penalty chosen
Error achieved
Against this run
1
0.01
1.05065
same choice
2
0.05
1.07182
different choice
3
0.01
1.04083
same choice
4
0.01
1.05129
same choice
5
0.1
1.08418
different choice
6
0.1
1.09890
different choice
7
0.05
1.06971
different choice
8
0.05
1.05886
different choice
9
0.1
1.08462
different choice
10
0.05
1.06294
different choice
11
0.1
1.11433
different choice
12
0.05
1.06079
different choice
Nothing about the data changed between these rows. Only which observations landed in which fold. A procedure whose answer moves when an arbitrary partition is re-drawn is reporting the partition as much as the data, and repeated cross-validation — averaging over several assignments rather than trusting one — is the standard remedy.
The error path, and how flat it is
Cross-validated error and its standard error at each penalty
Penalty
Cross-validated error
Its standard error
0.00001
1.02840
0.18304
0.0001
1.02839
0.18302
0.0005
1.02835
0.18295
0.001
1.02830
0.18286
0.005
1.02805
0.18220
0.01
1.02803
0.18144
lowest error
0.05
1.03368
0.17718
0.1
1.04486
0.17403
0.5
1.14884
0.16483
one standard error rule
1
1.32873
0.16226
Training error
1.00637
—
optimistic by 2.2%
A minimum in a flat basin is not a well-determined minimum. Several penalties here differ by less than one standard error, so which one wins is decided by noise, and the one-standard-error rule — take the largest penalty still within one standard error of the best, here 0.5 — exists precisely to stop that noise choosing the model.
The training error is 1.00637 against a cross-validated 1.02803 — lower by 2.2%, because the fit has already seen every row it is being scored on. That gap is the entire reason the procedure exists, and it widens sharply as a model gains flexibility.
The error at the chosen penalty is itself optimistic, because the same folds picked it. Reporting it as an estimate of future accuracy double-counts the tuning, and nested cross-validation — an outer loop for the estimate, an inner one for the choice — is the correction. This applies to every penalty chosen on the ridge, lasso and elastic net pages.
Twelve fold assignments Choice spread measured Full error path Training error beside it One-standard-error rule
What this tool shows
Re-drawing the folds twelve times on identical rows returns three distinct penalties spanning a tenfold range, while the error those choices achieve varies by 2.00%. Cross-validation is normally reported as though it delivers a number — the optimal penalty, chosen by the data. It delivers a random variable, and the randomness comes from an arbitrary partition nobody reports. The reason it usually does not matter is on the same screen: the error surface is nearly flat across that whole region, so the choices are interchangeable even though they look different.
K-fold cross-validation over a penalty path, with the standard error at every point
The whole procedure repeated across twelve different fold assignments
The spread of the chosen penalty against the spread of the error it achieves
The one-standard-error rule, and why a flat basin needs one
Training error beside the cross-validated error, with the optimism between them
Why the error at a tuned parameter is itself optimistic
Twelve fold assignments Choice spread measured Full error path Training error beside it
A cross-validated choice is a random quantity, not a determined one.
Updated 13 September 2026 · Works in any browser, no installation
Cross-validation estimates how well a model will do on data it has not seen, by repeatedly fitting on part of the data and scoring on the rest. It is the standard way to choose a tuning parameter, and the thing routinely left out of the report is that the answer depends on an arbitrary partition. Re-draw the folds and the chosen parameter moves; what usually does not move much is the accuracy it achieves, which is why the practice survives.
At a glance
Formula shown
The rows are split into k folds. For each fold the model is fitted on the other k−1 and scored on the held-out one, and the k scores are averaged. The standard error across those k scores gives the uncertainty at each parameter value; the one-standard-error rule takes the largest penalty whose error is still within one standard error of the best. Training error, computed on the same rows used to fit, is optimistic by construction.
Scenario support
Choosing a ridge or lasso penalty, choosing how many components to keep, comparing candidate models on the same data, estimating how a model will perform on new data, and any setting where a parameter is tuned rather than specified in advance.
Educational estimate
Planning support from the values you enter — not professional advice.
The number that moves
The preset is a hundred and fifty observations and four correlated predictors, cross-validated over a ridge path with five folds. The reference split picks a penalty of 0.01.
Nothing about the data then changes. Only which observations land in which fold. Twelve different assignments return: 0.01, 0.05, 0.01, 0.01, 0.1, 0.1, 0.05, 0.05, 0.1, 0.05, 0.1, 0.05. Three distinct values, spanning a tenfold range.
The error those choices achieve varies by 2.00%.
Both halves of that matter. The first says a reported optimal penalty is substantially a property of the partition, so “we chose λ = 0.01 by cross-validation” states a coincidence of the fold draw as a result. The second says it usually does not matter much, because the penalties in that range are close to interchangeable.
The path table shows why. The error is 1.028 at every penalty from 0.00001 through 0.01 — a flat basin, not a sharp minimum — and any point in a flat basin can win depending on noise. The procedure is not broken; it is being asked to distinguish things the data cannot distinguish, and it answers anyway.
The standard remedy is repeated cross-validation: run several fold assignments and average, rather than trusting one. It costs a multiple of the compute and removes most of this variability, and it is reported far less often than it should be.
Why held-out data is needed at all
The training error on this preset is 1.0064 and the cross-validated error is 1.0280 — optimistic by 2.2%. The fit was scored on rows it had already seen, and it does better on them than it will on new ones.
Two point two per cent is small because a ridge fit with four predictors on a hundred and fifty rows is not very flexible. The gap widens sharply with flexibility: a model with as many parameters as observations can reach zero training error and predict nothing. Training error is therefore not a weak estimate of future accuracy but a measure of a different thing entirely, and the gap between the two is what cross-validation exists to expose.
The same reasoning explains the fold count. More folds means each fit uses more data, so the error estimate has less bias — and the fits overlap more, so the estimate has more variance and costs more to compute. Five and ten are conventions that balance those, and leave-one-out sits at the extreme of the trade.
The error at a tuned parameter is optimistic too
This is the mistake worth being most careful about. Having used cross-validation to choose a penalty, the error at that penalty is no longer an unbiased estimate of anything — it is the minimum of several noisy estimates, and the minimum of noisy things is systematically low.
Reporting it as expected future accuracy double-counts the tuning. The bias grows with how many candidate values were searched, so a fine grid over two parameters is worse than a coarse grid over one, and a long model-selection exercise reported with its own cross-validated error can be substantially optimistic.
The correction is nested cross-validation: an outer loop that holds data out for the estimate, and an inner loop within each outer fold that does the tuning. It costs roughly the square of the compute and it is the honest procedure when a parameter was chosen rather than specified.
The same caution applies wherever a penalty or component count is tuned — ridge, lasso, elastic net, partial least squares — all of which report a single-loop figure that inherits this optimism.
The one-standard-error rule
The path reports a standard error beside each cross-validated error, computed across the folds. It is what makes the flatness of the basin visible: several penalties here differ by less than one standard error, which is to say the data cannot tell them apart.
The one-standard-error rule takes the largest penalty whose error is still within one standard error of the best. On a tie it prefers the more regularised model, on the argument that when the data is indifferent the simpler answer should win.
It matters more for methods that select than for those that only shrink, because there the difference between the two rules is the difference between reporting five predictors and three. It is a convention rather than a theorem, and it should be stated when used since it changes the model.
When random folds are the wrong folds
Everything here assumes rows are exchangeable — that any observation could sit in any fold. Where that is false, random folds produce an error estimate that is optimistic by far more than the 2.2% above.
Time series are the clearest case. A random fold lets the model train on Tuesday and Thursday to predict Wednesday, which is not a situation it will face. Forward-chaining splits, which always train on the past and test on the future, are the honest version.
Grouped data is the other common case: repeated measures on the same patient, several rows from the same school or firm. Random folds split those groups across the divide, so the model has seen something very like each test row and the estimate is badly inflated. Grouped folds, keeping every row from a group together, are the fix — the same reasoning that the design effect applies to standard errors.
Reporting it
Report the fold count and how the folds were formed. Random, stratified, grouped and time-ordered splits answer different questions, and a bare “5-fold cross-validation” does not say which was used.
Report the standard error with the error. It is what tells a reader whether the chosen point was distinguishable from its neighbours, and on a flat basin it is the difference between a result and a coin toss.
If a parameter was tuned, say whether the reported error came from a nested procedure. If it did not, say that too — it is a small admission and it lets a reader discount appropriately instead of taking an optimistic number at face value.
Method. Folds are formed by a seeded shuffle, so different seeds give genuinely different partitions rather than rotations of one — which is what makes the variability measurement meaningful rather than an artefact. The reference split and twelve further assignments each run the full path, and the spread of the chosen penalty is reported against the spread of the error it achieves. The suite confirms that the best error is the minimum on the path, that the one-standard-error penalty is never smaller than the best one and its error never lower, that training error never exceeds the cross-validated error so optimism is non-negative, and that every re-split chose a penalty that exists on the grid. That engine is verified on every change against 395 assertions. The count and the per-case breakdown are published on the formula verification page.
Related calculators
Where this goes next:
Ridge RegressionFit ridge regression across a penalty path chosen by cross-validation, with least squares beside it and the shrinkage applied to every coefficient shown.
Lasso RegressionFit a lasso across a penalty path with cross-validation, and see how often each selected predictor survives when the rows are resampled.
Elastic NetFit an elastic net across the mixing dial from pure lasso to pure ridge, with selection stability and cross-validated error computed at each end.
Partial Least SquaresBuild regression components against the outcome rather than from predictor variance, with the unsupervised decomposition computed beside it for comparison.
Principal Component RegressionRegress on principal components with each component's variance share printed beside its correlation with the outcome, and the error at every component count.
Design EffectThe design effect 1 + (m − 1)ρ with the effective sample size it implies, the ceiling of 1/ρ effective observations per cluster, and the marginal gain from each extra person.
An educational tool. A cross-validated error at a parameter chosen by the same procedure is optimistic; nested cross-validation is required for an unbiased estimate. Random folds assume rows are exchangeable and give badly inflated estimates for time series or grouped data, which need forward-chaining or grouped splits instead.