A two-proportion z-test on this data gives p = 0.06019 — not significant, and the usual advice is to keep testing. The Bayesian reading is that B is better with probability 0.96977, and that switching to B costs 0.0000782 in conversion rate if that judgement is wrong: 0.0078 percentage points. Whether to ship is a decision about that number, not about 0.05.
A 5.037% · B 6.286%
P(B beats A) = 96.977%
Choosing B costs 0.0000782 in conversion rate if A was really better — 0.0078 percentage points. Choosing A costs 0.0125678. The smaller of those is below the 0.001 threshold, so the decision is made.
P(B beats A)
96.977%
exact, not sampled
Expected loss, B
0.0000782
0.0078 percentage points
Expected loss, A
0.0125678
1.2568 percentage points
Expected lift
24.79%
B over A
Expected loss is the number to stop on, not the probability. It weighs how likely you are to be wrong against how much being wrong costs, and a 90% probability with a large possible gap can be a worse bet than a 97% probability with a small one.
Both posteriors
The posterior for each arm with its credible interval
Arm
Data
Posterior
Rate
95% credible
A
120 of 2400
Beta(121, 2281)
5.037%
4.199% to 5.947%
B
150 of 2400
Beta(151, 2251)
6.286%
5.351% to 7.291%
Overlapping intervals do not mean the arms are indistinguishable — these two overlap and B still wins 97.0% of the time, because the comparison integrates over both distributions rather than checking whether two ranges touch.
Exact, not simulated Loss, not just probability Margin of practical interest Peeking still costs you
What this tool shows
On the shipped preset a two-proportion z-test gives p = 0.06019 — not significant, keep testing. The same data gives a 96.977% chance that B is better and an expected loss from switching of 0.0078 percentage points. Those answer different questions, and only the second is the question a decision needs. P(B beats A) here is computed from a closed form rather than simulated, so the same input always gives the same answer.
P(B beats A) from the exact series for two Beta posteriors — no sampling, no seed
Expected loss for each arm, which is the number a stopping rule should use
A margin of practical interest: P(B beats A by at least x%) rather than by any amount
Both posteriors with credible intervals, and why overlapping intervals prove nothing
The frequentist p-value on the same data, for the comparison that is the point
A preset with a high probability and a loss too large to act on
Exact, not sampled Expected loss Practical margin Both posteriors
Bayesian testing does not make peeking free. It makes the cost of stopping explicit.
Updated 13 September 2026 · Works in any browser, no installation
A Bayesian A/B test gives the probability that one variant is better and the expected cost of being wrong. Both arms start from a prior, the data updates them into Beta posteriors, and the comparison integrates one against the other. That is a different object from a p-value: a p-value is the chance of seeing data this extreme if there were no difference at all, which is not the quantity anyone making a shipping decision actually wants.
At a glance
Formula shown
With a Beta(1,1) prior, arm A becomes Beta(1+x_A, 1+n_A−x_A) and likewise for B. Then P(B > A) = Σᵢ₌₀^{a_B−1} B(a_A+i, b_A+b_B) / [(b_B+i)·B(1+i, b_B)·B(a_A, b_A)], a sum that terminates after a_B terms. Expected loss from choosing B is E[max(0, A − B)], which follows from the same series applied to the first moments — so both numbers are exact rather than estimated.
Scenario support
Deciding whether to ship a variant, running tests where traffic arrives continuously and a fixed sample size is impractical, comparing conversion rates with very different volumes, and any test where the cost of a wrong decision is known and a p-value threshold is not the right rule.
Educational estimate
Planning support from the values you enter — not professional advice.
p = 0.060, and a 97% chance B is better
The two frameworks agree on clear results and disagree exactly where the decision is hard. The first preset sits there deliberately.
120 of 2400 against 150 of 2400. A two-proportion z-test gives z = 1.8794 and p = 0.06019 — not significant.
The posterior probability that B is better is 0.96977, and the expected cost of switching to B if A was really better is 0.0078 percentage points.
Neither number is wrong; they answer different questions. p = 0.060 is the chance of a gap this large if the variants were identical. 0.97 is the chance that B is better given what was observed. Only the second can be weighed against a business cost.
On the fourth preset both agree: p = 0.0001016 and P = 0.99995. The frameworks part company in the borderline zone, which is the only zone where the answer matters.
Stop on the loss, not the probability
A probability alone is a poor stopping rule, and the second preset shows why in one line.
8 of 120 against 14 of 120 gives a 90.62% chance that B is better. That sounds like enough.
The expected loss from picking B is 0.0016723 — 0.17 percentage points, against 0.0078 on the first preset where the probability was only slightly higher.
The difference is how far apart the arms could still be. With 120 visitors each, the posteriors are wide, so the 9% chance of being wrong comes with a large possible cost.
Expected loss combines both. It is the probability of being wrong times how wrong, integrated, and a threshold on it — 0.001 here — is a rule in the units of the thing you care about rather than in units of nothing.
The question is rarely “better at all”
Shipping a variant costs something: engineering time, risk, the option to test something else. A difference of zero is almost never the threshold.
The third preset asks whether B beats A by at least 5% in relative terms. The probability falls from 0.96977 to 0.92841.
That gap is the entire difference between “B is better” and “B is worth shipping”. A test can be decisive about the first and undecided about the second.
The margin should be set before the test runs, from what the change would cost to implement and maintain, not chosen afterwards from the number that gives the answer you want.
A region of practical equivalence is the mirror image: the band inside which the two arms are effectively the same, and a test that ends inside it has concluded something useful rather than failed.
Bayesian testing does not make peeking free
This is the most common overclaim made for Bayesian A/B testing, and it is worth being precise about what is and is not true.
What is true: the posterior is valid whenever you look at it. It does not depend on a stopping rule the way a p-value does, so checking daily does not invalidate the number.
What is not true: that stopping when the probability crosses a threshold has no cost. It does. Watching a random walk and stopping when it wanders past a line inflates how often you stop on noise.
The expected-loss rule is what actually controls this, because early in a test the posteriors are wide and the loss stays large no matter how the probability wanders.
A minimum sample size still belongs in the design. Bayesian machinery changes what you compute, not the fact that a hundred visitors cannot settle a 5% difference.
What the flat prior is doing
Both arms here start from Beta(1, 1), the uniform prior. It is not neutral in every sense and it is worth knowing where it shows.
It behaves like one success and one failure already observed. On 2400 visitors that is 0.08% of the evidence and invisible; on 20 visitors it is 9% and visible.
It pulls every estimate toward 50%, which for a conversion rate near 5% means upward. Beta(1,1) with 0 of 30 gives a posterior mean of 3.1%, not 0.
For conversion rates a better default is a weak prior centred where rates usually sit — something like Beta(2, 40) for a 5% rate, worth about 42 visitors of evidence.
The honest test is to run it both ways. If the prior changes the decision, the data has not settled the question and the prior is doing work it should be asked to justify.
Overlapping intervals prove nothing
The most common way to misread the table on this page is to compare the two credible intervals and conclude that overlap means no difference.
On the first preset the intervals overlap — A runs to 5.947% and B starts at 5.351% — and B still wins 97.0% of the time.
The comparison integrates one whole posterior against the other. It asks how often a draw from B exceeds a draw from A, which is a far weaker requirement than the intervals being disjoint.
Non-overlapping intervals imply a difference; overlapping ones imply nothing. The implication runs one way only, and the same error is made constantly with frequentist confidence intervals.
Which is why the probability is computed rather than eyeballed, and why the intervals are on this page as description rather than as a test.
Reporting a Bayesian test
Five items, and the second is the one that makes the result a decision rather than a number.
Give both arms’ raw counts. Conversions and visitors, not just rates — every number here follows from them.
Give the expected loss, not only the probability. A 90% probability with a large loss and a 97% probability with a small one call for opposite decisions.
State the prior. Flat is a choice, and on small samples it is a visible one.
State the margin and the loss threshold, and say they were set in advance. Both are business decisions, and both can be chosen after the fact to produce any answer.
And say how often you looked. The posterior is valid at every look; the stopping behaviour still needs describing.
Sources and methodology
References for the closed form and the decision rule.
Method. P(B beats A) is computed from the terminating series for two Beta posteriors rather than by sampling, which is why the same input always returns the same answer — a simulated figure that moves between runs is not something to make a shipping decision on. The suite checks that series against a 40,000-draw simulation at four very different configurations, including one at n = 10,000 per arm and one at n = 12, and asserts the identity P(B>A) + P(A>B) = 1 to ten decimal places. Expected loss comes from the same series applied to the first moments, so it is exact too, and the suite asserts that whichever arm is more likely to be better always carries the smaller loss — a consistency the two quantities have no reason to satisfy unless both are right. The margin probability is the one number here obtained by numerical integration, over A’s posterior against B’s tail. Both arms use a flat Beta(1, 1) prior, which the page names rather than hides. That engine is verified on every change against 106 assertions. The count and the per-case breakdown are published on the formula verification page.
Related calculators
Where this goes next:
Credible IntervalBayesian credible intervals for a proportion with both the highest-density and equal-tailed versions and the prior's weight reported.
Two-Proportion Z-TestCompares two rates with the pooled z-test and a Newcombe interval for the difference, and shows that z² equals the 2×2 chi-square statistic exactly — so the “z-test or chi-square” question has no content.
Bayes FactorBayes factors for a proportion in closed form with the exact binomial p-value beside them and Jeffreys' bands in both directions.
Posterior ProbabilitySequential Bayesian updating with the posterior after every batch, the prior's weight reported, and grouping-independence demonstrated.
Conversion RateConversion rate and revenue from traffic and orders, with what a 10% or 25% lift would actually be worth.
Sample SizeResponses needed for a target margin of error, with the finite-population correction and a table of the whole cost curve — because n scales with 1/margin², so the last point of precision costs more than the first ten.
An educational tool. A Bayesian analysis does not make repeated looks free — the posterior is valid whenever you check it, but stopping the moment a probability crosses a line still inflates how often you stop on noise, which is what the expected-loss threshold exists to control. The flat prior used here is a choice, and on small samples it visibly pulls estimates toward 50%.
Launched Bayesian A/B testing with P(B beats A) from the terminating Beta series rather than by sampling, so the same input always returns the same answer.
Checked that series against a 40,000-draw simulation at four configurations and against the identity P(B>A) + P(A>B) = 1, which it satisfies to ten decimal places.
Made expected loss the headline stopping rule rather than the probability, with a preset where 90.62% confidence carries a loss too large to act on.
Added a margin of practical interest: on the shipped data P(B beats A) is 0.96977 and P(B beats A by 5%) is 0.92841.
Printed the frequentist p-value on the same data — 0.06019 where the posterior probability is 0.96977.
Add this calculator to your site
Responsive embed — and private: nothing your visitors type leaves their browser.