An ad with 9 purchases isn't a winner. It's a coin flip..

Last updated: 23 August 2026

You have two ads in the same campaign, same seven days, roughly equal delivery: Ad A shows 9 purchases, Ad B shows 5. Somewhere between that screen and the scale button sits the question nobody in your ads manager will answer for you, which is whether that result clears the bar for meta ads statistical significance or only looks like it does.

The dashboard paints an 80% improvement in green (illustrative numbers throughout this post, never client data). Your cursor is already moving.

A 9-to-5 purchase split between two equally delivered ads is statistically indistinguishable from a coin flip (illustrative arithmetic).

Give it ninety seconds first. The answer is arithmetic, you can rerun every step of it in a spreadsheet, and it is not what most published testing guides tell you.

Is a 9-5 split actually a winner?

Start with a better question. Not "which ad converted better," which the dashboard has already answered in the most flattering way available. Ask instead: given that 14 purchases happened across the two ads, how surprising is a 9-5 split if the two ads are exactly equally good?

That reframe changes the method. If both ads are genuinely equal and both got equal delivery, each individual purchase was equally likely to land on either one. Fourteen purchases splitting between two equally good ads follows the same distribution as flipping a fair coin fourteen times. Not a figure of speech: the same distribution, formalised as the conditional test for two Poisson rates (Przyborowski & Wilenski, 1940), equivalent to Fisher's exact test conditioned on the total.

So count the outcomes. Fourteen coin flips produce 2^14 = 16,384 equally likely sequences. The sequences giving nine or more heads number 2,002 + 1,001 + 364 + 91 + 14 + 1 = 3,473. Divide: 3,473 / 16,384 = 0.212. Then double it, because Ad B could have been the one running ahead: 0.424.

If two ads received roughly equal delivery and produced 14 purchases between them, a 9-to-5 split occurs about 42% of the time even when the two ads are exactly equally good (illustrative arithmetic, binomial n=14, p=0.5).

The coin flip in the headline is not a metaphor. It is the model.

What split would actually prove something?

The obvious follow-up: fine, 9-5 is noise, so what would not be? With 14 total purchases on the table, here is every split that gets anywhere near the conventional 5% significance bar.

Illustrative arithmetic in the table below: two-sided p-values computed as 2 x P(X >= k) for a binomial with n=14 and p=0.5.

Look at the 11-3 row, because it is the one that stings. Eleven purchases against three feels decisive. It is a p-value of 0.057, which misses the bar you were implicitly claiming to clear.

With 14 total purchases split across two ads (illustrative), no result short of a 12-to-2 split reaches the conventional 5% significance threshold.

Most operators have never scaled anything close to 12-2. That is not a personal failing. It is the sample size.

You can run this check on your own account in two minutes.

1. Add up the total conversions across both ads for the exact window you are comparing, not the campaign lifetime and not a window you picked after seeing the result.
2. Read your split against a table like the one above, or compute the two-sided p-value yourself: sum the binomial probabilities for your count or anything more lopsided, then double it.
3. Compare that number to a bar you set before you looked, not to whichever figure the ads manager decided to highlight in green.

Step three is the one people skip, and it is the one that does the work. A threshold chosen after seeing the data is not a threshold. It is a rationalisation with a decimal point.

Doesn't the reported conversion rate settle it?

Not on its own, and this is where a second piece of arithmetic earns its keep.

A reported conversion rate is a point estimate. What you want is the range of true rates that could plausibly have produced it. The tool for that is the Wilson score interval (Wilson, 1927).

Do not use the textbook Wald interval that most free significance calculators run on, shaped like p ± 1.96 x sqrt(p(1-p)/n). Brown, Cai and DasGupta (2001) found its coverage erratic and unreliable in exactly this regime of small counts and proportions near zero, concluding that "common textbook prescriptions regarding its safety are misleading and defective." That is a critique of a method, not of any one tool. It also means the calculator on page one of your search results is probably using the weaker estimator on the data least suited to it.

Wilson, in spreadsheet form, where p = successes / n and z = 1.96:

centre     = (p + z^2/(2n)) / (1 + z^2/n)
        half-width = (z / (1 + z^2/n)) * SQRT( p*(1-p)/n + z^2/(4n^2) )
        interval   = centre ± half-width

Now feed it the two ads, each with 1,000 clicks in the test window.

Ad A, 9 purchases from 1,000 clicks (illustrative): reported rate 0.90%, 95% Wilson interval 0.47% to 1.70%.

Ad B, 5 purchases from 1,000 clicks (illustrative): reported rate 0.50%, 95% Wilson interval 0.21% to 1.17%.

Nine purchases from 1,000 clicks (illustrative) is a 0.9% conversion rate with a 95% Wilson confidence interval of 0.47% to 1.70%, a range that is nearly four times as wide at the top as at the bottom.

Set those ranges beside each other and they overlap across most of their width. Ad A's true rate might be 0.5%. Ad B's might be 1.1%. The ordering you can see in the dashboard is not evidence of the ordering that exists in reality.

If the ranges overlap, does that mean there's no difference?

No, and this is the mistake people make immediately after learning the last one. "The intervals overlap, so there is no difference" is not a valid test. It is a visual habit, and it is wrong in both directions.

The correct object is a confidence interval on the difference between the two rates. Newcombe (1998) compared eleven methods for exactly this and found his hybrid score method, method 10, performs well at any sample size. It combines the two Wilson intervals you already have:

lower = (p1 - p2) - SQRT( (p1 - L1)^2 + (U2 - p2)^2 )
        upper = (p1 - p2) + SQRT( (U1 - p1)^2 + (p2 - L2)^2 )

L1, U1 and L2, U2 are the Wilson bounds from above. Run it on our two ads and the difference in conversion rate sits somewhere between -0.39 and +1.25 percentage points, with 95% confidence (illustrative, Newcombe hybrid score method).

Read that range slowly. It is consistent with Ad A being somewhat worse than Ad B. It is equally consistent with Ad A being roughly two and a half times better. Both conclusions live inside the same interval, drawn from the same numbers that produced the confident green badge.

Which makes the point estimate the least informative summary available. "80% better" is one number pulled from the middle of a range wide enough to contain the opposite conclusion.

What would it actually take to know for sure?

Here is the question the guides answer badly, and the reason they contradict each other.

Sample size has no universal answer, because it depends on the size of the effect you want to be able to detect. A test powered to catch a huge difference needs far less data than one powered to catch a small one. Almost no published rule of thumb says which it is describing.

Using the standard two-proportion sample size formula (Fleiss, Levin & Paik, 2003, ch. 4), at 80% power and 5% two-sided significance with equal arms:

n per arm = [ z(a/2)*SQRT(2*pbar*(1-pbar)) + z(b)*SQRT(p1*(1-p1) + p2*(1-p2)) ]^2 / (p1 - p2)^2

with z(a/2) = 1.96, z(b) = 0.8416, and pbar the average of p1 and p2. From a 1.0% baseline conversion rate:

Detecting a 20% relative improvement in conversion rate from a 1.0% baseline (illustrative), at 80% power and 5% significance, requires roughly 42,700 clicks per ad, or about 427 purchases per ad.

That table also resolves the contradiction running through every testing guide you have read. The guides recommending twenty to fifty conversions are assuming, without saying so, that you only care about enormous effects. The guides recommending a hundred or more are assuming something smaller. Neither is wrong. Both are incomplete, because neither names the effect size it is powered for.

Which gives you a filter for every threshold you will ever be handed again: a rule of thumb that does not name an effect size is not a rule, it is a guess with a number attached.

Why doesn't more data just fix this?

Because two problems survive a perfectly powered test, and one of them belongs to ad platforms specifically.

Peeking. You open Ads Manager every morning, and that habit has a statistical cost. Armitage, McPherson and Rowe (1969) established the mechanism: if significance tests at a fixed level are repeated at stages during the accumulation of data, the probability of a significant result when the null hypothesis is true rises above the nominal level. Checking daily and stopping the moment something crosses the line is not neutral observation. It is a decision rule, and a permissive one. If you want a method that stays valid under continuous monitoring, Johari et al. (2022) is the modern treatment.

False discovery. Even among results that clear the bar honestly, a meaningful share are not real. According to a 2022 study published in Management Science, between 18% and 25% of A/B test results declared significant at the 5% level are false discoveries. That study covered 4,964 effects from 2,766 experiments on a commercial testing platform, found roughly 70% of tested effects were true nulls, and put the false discovery rate at the 10% level between 28% and 37% (Berman & Van den Bulte, 2022).

Divergent delivery. This one should change how you read every creative test you have run. Ad platforms do not show two creatives to the same people. They optimise delivery for each one separately, sending each ad to a different and undetectable mix of users, including inside the platform's own test tooling. Exposure is not random, so what you measure is the creative's effect tangled with the targeting's effect, with no way to separate them from outside.

According to a 2025 study published in the Journal of Marketing, ad platform algorithms show each creative to a different mix of users, so an ad comparison confounds the effect of the creative with the effect of the targeting, and can reverse which ad looks like the winner (Braun & Schwartz, 2025).

Sit with that last clause. The distortion is not only in the magnitude. It can flip the sign. The losing ad can be the better ad.

None of this is fringe. Lewis and Rao (2015) showed how enormous advertising experiments must be before their returns can be measured reliably, and Gordon et al. (2019), across 15 experiments and 500 million user observations on Facebook, found observational methods often fail to reproduce what the randomised experiments found.

Does Meta's own test tool solve this for you?

Partly, and it is worth knowing which part.

Meta's lift study documentation states that breakdown combinations must have at least 100 conversions from test and control groups combined before results will display, and that partners are warned when a study does not meet the sufficient power threshold (Meta for Developers, lift studies documentation). That is a real guardrail against rendering a confident-looking breakdown on almost nothing.

The split testing documentation is more interesting for what it omits. It advises establishing a confidence level before testing, and notes that tests with larger reach, longer schedules, or higher budgets tend to deliver more statistically significant results. It specifies no minimum conversion count and no minimum budget (Meta for Developers, split testing documentation). The widely repeated claims that Meta recommends a specific number of conversions per variation do not appear there.

Worth noting too: platform tooling and published guides use different confidence bars for different test types, which is part of why the "right number" of conversions keeps changing depending on which source you read.

So the direct answer is no. The tool manages some display thresholds. It does not pick your effect size, it does not stop you peeking, and it cannot undo divergent delivery, because divergent delivery is how the platform works, not a setting inside it.

So what do you do instead?

Here is the turn, and it is more useful than it sounds.

If you cannot reach significance at your spend, and the sample size table says most DTC accounts cannot, stop calling it a test. You are making bets, not running experiments.

Bets are fine. Every good operator makes them. What is not fine is making a bet and reporting it as an experiment, because that is how an account ends up scaling on a coin flip and then quietly wondering why the winner faded. Bets made deliberately beat experiments performed badly.

So replace false certainty with discipline that holds at your volume:

• Write the hypothesis down before you launch. One sentence naming what you believe and why. A test with no stated belief cannot be wrong, so it cannot teach you anything either.
• Pre-commit to the signal you will treat as directional. Decide the number before you see the number. Cheapest upgrade available to most accounts.
• Change one variable at a time. One-variable discipline is what makes a result interpretable if it ever does accumulate.
• Judge concepts across a portfolio over time. Compare the pattern across many attempts instead of ranking two ads on fourteen purchases.

That last point needs a boundary stated plainly: running more creative does not create statistical significance. Nothing does, short of the sample sizes in that table. What accumulating deliberate bets gives you is directional evidence that compounds, which beats a single underpowered verdict delivered with false confidence. Results vary, and no cadence changes the arithmetic. If your winner faded after you scaled it, regression to the mean under creative fatigue explains part of why, and how many new ads to launch each week covers the volume side.

A written hypothesis and a pre-committed success signal are two of the named contents of a CreatStrat brief, alongside the hooks, the script, the format, and the audience. CreatStrat is an automated creative strategy product: a software subscription that researches your brand, your customers, your market, and your connected ad performance, then delivers production-ready briefs every week. You produce the ads. CreatStrat handles what to make and why.

And if your tests are inconclusive at your volume, the answer is better concepts, not more math. Statistical sophistication does not rescue an underpowered account. Better starting concepts, chosen from research instead of guesswork, change the odds before the test runs.

Frequently asked questions

How many conversions do you need for statistical significance?

It depends on the effect size you want to detect. Detecting a 20% lift from a 1% baseline takes roughly 427 purchases per ad (illustrative); detecting a 10% lift takes about 1,630. There is no single universal number, which is why published thresholds disagree.

Is 9 purchases enough to call an ad a winner?

No. Two ads showing 9 and 5 purchases produce that split, or a more lopsided one, about 42% of the time by chance alone, even when the ads are exactly equally good (illustrative, binomial n=14). That is a coin flip, not a result.

What is a good confidence level for ad testing?

The conventional academic bar is 95%, meaning a 5% false-positive rate. Platform tooling and published guides use different confidence bars for different test types, which is part of why the thresholds you read about vary so widely between sources.

Why does my Facebook or Meta ad test never reach significance?

Most DTC accounts do not get enough conversions per ad to detect anything except a very large effect. Checking results daily and platform delivery bias both make it worse, even in accounts where conversion volume is adequate.

Does Meta's built-in A/B test tool tell you if a result is significant?

Partially. Its lift studies require at least 100 combined conversions before displaying a breakdown result. It does not remove the need to think about effect size, repeated checking, or delivery bias, all of which sit outside the tool.

If two confidence intervals overlap, does that mean there's no real difference?

Not necessarily. Overlapping intervals are not a valid significance test on their own. The correct check is a confidence interval on the difference between the two rates, using a method built for it, such as Newcombe's hybrid score approach.

The short version

Fourteen purchases is fourteen coin flips. Nothing short of 12-2 clears the bar at that volume. Detecting a modest real improvement takes hundreds of purchases per ad, and even a perfectly powered test still has peeking and divergent delivery working against it.

So the leverage is not downstream in the measurement. It is upstream, in what you decide to make.

Get 3 free ad ideas. No card. No call. Get three creative opportunities within 48 hours. You get three creative opportunities, one audience gap, and one example concept, built from research on your brand, your customers, and your market. A better starting point, not a shortcut around the math.

A note on method

Every citation below was verified against the publisher record, DOI, PMID, or author institutional listing during research for this post. Every worked figure was computed independently and cross-checked against the cited method. All ad numbers are illustrative, invented for teaching, and are not customer data. You can reproduce every calculation on this page in a spreadsheet using the formulas as printed.

Sources

1. Braun, M., & Schwartz, E. M. 2025. Where A/B testing goes wrong: How divergent delivery affects what online experiments cannot (and can) tell you about how customers respond to advertising. Journal of Marketing. 89(2):71-95. doi:10.1177/00222429241275886.
2. Berman, R., & Van den Bulte, C. 2022. False discovery in A/B testing. Management Science. 68(9):6762-6782. doi:10.1287/mnsc.2021.4207.
3. Lewis, R. A., & Rao, J. M. 2015. The unfavorable economics of measuring the returns to advertising. The Quarterly Journal of Economics. 130(4):1941-1973. doi:10.1093/qje/qjv023.
4. Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. 2019. A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science. 38(2):193-225. doi:10.1287/mksc.2018.1135.
5. Wilson, E. B. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association. 22(158):209-212. doi:10.1080/01621459.1927.10502953.
6. Brown, L. D., Cai, T. T., & DasGupta, A. 2001. Interval estimation for a binomial proportion. Statistical Science. 16(2):101-133. doi:10.1214/ss/1009213286.
7. Agresti, A., & Coull, B. A. 1998. Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician. 52(2):119-126. doi:10.1080/00031305.1998.10480550.
8. Newcombe, R. G. 1998. Interval estimation for the difference between independent proportions: Comparison of eleven methods. Statistics in Medicine. 17(8):873-890. doi:10.1002/(SICI)1097-0258(19980430)17:8<873::AID-SIM779>3.0.CO;2-I. PMID: 9595617.
9. Armitage, P., McPherson, C. K., & Rowe, B. C. 1969. Repeated significance tests on accumulating data. Journal of the Royal Statistical Society Series A (General). 132(2):235-244. doi:10.2307/2343787.
10. Johari, R., Koomen, P., Pekelis, L., & Walsh, D. 2022. Always valid inference: Continuous monitoring of A/B tests. Operations Research. 70(3):1806-1821. doi:10.1287/opre.2021.2135.
11. Przyborowski, J., & Wilenski, H. 1940. Homogeneity of results in testing samples from Poisson series: With an application to testing clover seed for dodder. Biometrika. 31(3/4):313-323. doi:10.1093/biomet/31.3-4.313.
12. Clopper, C. J., & Pearson, E. S. 1934. The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika. 26(4):404-413. doi:10.1093/biomet/26.4.404.
13. Fleiss, J. L., Levin, B., & Paik, M. C. 2003. Statistical methods for rates and proportions (3rd ed.). Wiley. ISBN 9780471526292. (Chapter 4: sample-size formulas for detecting a difference between two proportions.)
14. Kohavi, R., Tang, D., & Xu, Y. 2020. Trustworthy online controlled experiments: A practical guide to A/B testing. Cambridge University Press. ISBN 9781108724265.
15. Meta for Developers, Marketing API. Conversion lift / lift studies documentation. https://developers.facebook.com/docs/marketing-api/guides/lift-studies/
16. Meta for Developers, Marketing API. Split testing documentation. https://developers.facebook.com/docs/marketing-api/guides/split-testing/