One of the most common questions in A/B testing sounds simple:
How long should an A/B test run?
Seven days? Two weeks? A month?
Unfortunately, there is no universal number.
An experiment should not stop because a calendar says enough time has passed. It should stop when you have collected enough evidence to make a useful decision.
That depends on how much traffic reaches the experiment, how frequently visitors convert, how different the variants really are, how much uncertainty you are willing to accept, and whether the test has covered a representative period of normal user behavior.
A high-traffic ecommerce store may collect enough data in a few days to evaluate a meaningful change. A B2B landing page with a handful of weekly conversions may need several weeks or longer.
This is why rules such as “always run a test for two weeks” are appealing but incomplete.
Two weeks can be excessive for one experiment and nowhere near enough for another.
The better question is:
What needs to happen before I can trust the result enough to act on it?
In this guide, we will explain what determines A/B test duration, why stopping too early is dangerous, when continuing a test stops being useful, and how to decide when an experiment has collected enough evidence.
If your main concern is whether you have enough traffic to experiment at all, read our guide to A/B testing for low-traffic websites.
Why There Is No Universal A/B Test Duration
Imagine two websites.
The first is a popular ecommerce store. Its product page receives 50,000 visits per week and generates thousands of purchases.
The second belongs to a consulting company. Its main service page receives 1,000 visits per month and generates around 25 leads.
Both companies create an A/B test.
If both experiments run for 14 days, they will finish with completely different amounts of information.
The ecommerce test may have tens of thousands of observations and hundreds or thousands of conversions. The consulting experiment may have only a few hundred visits and perhaps a dozen conversions.
The amount of calendar time is identical.
The amount of evidence is not.
That is why experiment duration should be considered as a consequence of data collection, not as the primary target.
Traffic determines how quickly visitors enter the experiment. Conversion rate determines how quickly meaningful events accumulate. The difference between the alternatives determines how easy it is to distinguish one from another.
A large effect generally becomes visible sooner.
If version A converts at 3% and version B converts at 8%, the difference is substantial.
If A converts at 3% and B converts at 3.1%, determining whether that tiny difference is real requires far more information.
The duration of an experiment is therefore the result of several interacting variables rather than one predefined number of days.
Why Stopping Too Early Is So Tempting
The beginning of an A/B test can be exciting.
You launch two versions of a page, return to the results dashboard after a few hours, and version B appears to be winning by 60%.
It is tempting to stop immediately.
After all, why continue showing visitors the apparently inferior version?
The problem is that early conversion rates are extremely unstable.
Imagine that the first five visitors assigned to B produce two conversions, while the first five visitors assigned to A produce none.
Version B has a 40% conversion rate. Version A has 0%.
That looks like an enormous difference.
After 100 visitors per version, A may have four conversions and B may have five. The apparent advantage has almost disappeared.
After 500 visitors, A may actually be ahead.
Nothing unusual happened. The early sample was simply too small.
Random events have a much larger effect when very few observations exist. Each additional conversion can dramatically change the apparent conversion rate.
This is one of the reasons A/B testing platforms use statistical confidence rather than simply declaring whichever variant currently has the highest conversion rate to be the winner.
Nelio A/B Testing uses the number of visits and conversions for each variant to calculate confidence. By default, it will not start computing results before an experiment has received at least 100 page views, and it will not declare a winning variation before the test reaches a minimum of 500 page views. These thresholds can be configured, but they exist to reduce the risk of drawing conclusions from tiny samples.
Statistical Confidence Is More Important Than the Calendar
When people ask how long to run an A/B test, what they often really want to know is:
When can I trust the winner?
Statistical confidence helps answer that question.
Nelio A/B Testing uses a G-test to compare the number of visitors and conversions received by each variant. From that comparison, it calculates a confidence value indicating how strongly the observed difference supports one variant outperforming another.
A commonly used threshold is 95% confidence.
Nelio uses 95% as the recommended default for highlighting a clear winner, although the required confidence can be configured.
That does not mean 95% is a magical value.
A business willing to accept more uncertainty may act at a lower threshold. A highly consequential experiment may justify waiting for stronger evidence.
For example, changing the text of a newsletter CTA and changing the price of a flagship product do not carry the same level of risk.
The more expensive a wrong decision would be, the more evidence you may want before implementing it.
Statistical confidence therefore provides a better stopping criterion than an arbitrary number of days, but it should still be interpreted alongside business context.
Nelio A/B Testing
By combining A/B tests made with Nelio, we doubled the revenue of webshops in just 12 months. I would recommend Nelio A/B testing to anyone with a WordPress site who wants to start A/B testing. It is definitely the best A/B testing plugin with the best price/quality for WordPress and WooCommerce.
Pieter Boonstra
Sample Size Still Matters
Confidence should not be considered independently from sample size.
This is particularly important at the beginning of an experiment.
A very unusual sequence of conversions can sometimes create an apparently strong difference when only a small number of visitors have participated.
For that reason, minimum sample thresholds act as an additional safeguard.
Nelio separates two concepts.
The first is the minimum amount of traffic required before it begins calculating meaningful results. The default is 100 page views.
The second is the minimum sample size required before the tool can declare a winner. The recommended default is 500 page views.
These values should not be interpreted as saying that 500 visits are sufficient for every experiment.
They are minimum safeguards.
An experiment comparing two similar conversion rates may need thousands or tens of thousands of observations before the difference becomes convincing.
A test involving a dramatic improvement may reach a useful conclusion much sooner.
The correct amount depends on what actually happens during the experiment.
Conversions Matter More Than Visits Alone
Visits create opportunities to observe behavior.
Conversions create the evidence you actually compare.
Consider two experiments that each receive 5,000 page views.
The first measures clicks on a prominent CTA and records 1,000 conversions.
The second measures completed purchases of an expensive product and records only 12.
Both experiments have the same number of page views, but the statistical information they contain is very different.
The first experiment has a large number of conversion events with which to compare the variants.
The second is still highly sensitive to a few individual purchases.
This is why conversion frequency has such a strong influence on how long a test needs to run.
A test measuring a common event such as a CTA click may accumulate information quickly.
A test measuring a rare event such as an enterprise purchase may need much longer.
Whenever possible, choose a conversion action that reflects the real business objective while occurring frequently enough to make experimentation practical.
For example, a SaaS company may test trial registrations rather than waiting for long-term renewals. A lead-generation site may measure form submissions rather than closed contracts.
The metric still needs to matter.
Optimizing an easy click that has almost no relationship with revenue can produce a statistically convincing answer to the wrong question.
Nelio A/B Testing lets you define conversion actions such as page visits, clicks, purchases, and other events, and attributes each conversion to the variant the visitor experienced.
Why You Should Usually Cover a Full Business Cycle
Statistical confidence is important, but there is another reason not to stop an experiment after two unusually good days.
Visitor behavior changes over time.
Monday traffic may behave differently from weekend traffic. Visitors arriving during working hours may differ from those browsing at night. Email campaigns, paid ads, newsletters, promotions, and social media can all change the composition of your audience.
Imagine that you start a test on Friday afternoon.
By Sunday evening, version B is clearly ahead.
If your business receives mostly consumer traffic on weekends but mostly professional traffic during the week, those three days may not represent the audience you normally serve.
Stopping the test would answer a narrower question:
Which version worked better for the visitors who arrived this weekend?
That may not be the question you intended to ask.
As a practical rule, allow an experiment to cover at least one complete business cycle whenever possible.
For many websites, that means including all seven days of the week.
For some businesses, one week may not be enough.
An ecommerce store with strong payday effects, a B2B company with monthly procurement cycles, or a travel website affected by seasonality may need a longer period before the traffic mix is representative.
This is another reason “seven days” cannot be a universal answer.
The relevant cycle depends on the business.
Be Careful With Marketing Campaigns
Traffic quality can change dramatically when a campaign starts or ends.
Suppose a normal landing page receives mostly organic traffic.
Halfway through an experiment, the marketing team launches an email campaign that sends thousands of existing customers to the page.
Those visitors may behave very differently from the search traffic that dominated the first half of the experiment.
The test is still technically valid if traffic is distributed consistently between variants, but interpreting the final result becomes more complicated.
Perhaps version B works better for existing customers and version A works better for first-time visitors.
A single overall conversion rate can hide that difference.
Campaigns are not a reason to avoid experimentation. They are a reason to understand the audience entering the test.
If the campaign is part of the normal marketing strategy, covering it may actually make the experiment more representative.
If it is an unusual one-day promotion that will never happen again, you should be cautious about letting it dominate the decision.
Audience segmentation can help when different visitor groups have genuinely different needs.
Do Not Stop a Test Because You Like the Result
One of the easiest ways to unintentionally bias an A/B test is to repeatedly check the results and stop when they show what you hoped to see.
You create version B because you believe it will win.
On day three, B is behind.
You continue.
On day five, B is still behind.
You continue.
On day eight, B jumps ahead.
You stop.
Without realizing it, you have created an asymmetric stopping rule.
The alternative was allowed unlimited opportunities to recover when it was losing, but the experiment ended the first time it looked successful.
This dramatically increases the risk of acting on random fluctuations.
The stopping criterion should therefore be defined before or independently from the current result.
For example, you might decide that the experiment will stop when it reaches an adequate sample size and the desired confidence threshold, or after a predefined amount of traffic if no meaningful difference emerges.
Nelio A/B Testing supports automatic end conditions based on duration, total page views, or statistical confidence in certain plans. An experiment can also be stopped manually.
Should You Always Wait for 95% Confidence?
Not necessarily.
The 95% threshold is widely used because it provides a reasonable balance between certainty and practicality.
But business decisions are rarely purely statistical.
Imagine two experiments.
In the first, you are comparing two versions of a secondary blog CTA. The alternative appears to produce a meaningful improvement with 92% confidence.
The cost of choosing incorrectly is relatively small.
You may decide that the evidence is sufficient.
In the second experiment, you are considering a major pricing change that could affect hundreds of thousands of euros in annual revenue.
You may want considerably stronger evidence before implementing it.
The right confidence threshold depends partly on how costly a false positive would be.
Nelio lets you configure the required confidence level, and its documentation notes that businesses comfortable with more risk may act earlier, while those seeking greater certainty can wait longer.
The statistical report informs the decision.
It does not replace business judgment.
What If a Test Never Finds a Winner?
Some experiments simply do not produce a meaningful difference.
You may run a test for several weeks, collect substantial traffic, and watch both conversion rates remain extremely close.
That does not mean the experiment failed.
It may mean that the change does not matter very much.
Suppose the original button says “Get Started” and the alternative says “Start Now.”
After thousands of visits, both versions continue converting at almost exactly the same rate.
You could keep the experiment running for another six months in the hope that one eventually reaches the confidence threshold.
But ask yourself what you are trying to learn.
If the difference is so small that an enormous amount of traffic is needed to detect it, the practical value of the improvement may also be very small.
At some point, the opportunity cost of continuing becomes more important than obtaining a definitive winner.
The traffic currently being consumed by this inconclusive test could be used to evaluate a much stronger hypothesis.
An experiment can therefore end with a useful conclusion even when there is no winner:
This change does not appear important enough to pursue.
That is a legitimate learning.
How Long Is Too Long?
There is no exact maximum duration either.
A test running for two months is not automatically invalid.
The question is whether the experiment is still testing the same conditions.
The longer an experiment runs, the greater the chance that something else changes.
You may update your theme, change a marketing campaign, introduce a new product, alter pricing, modify navigation, or experience seasonal changes in visitor behavior.
If the underlying website or audience changes substantially during the test, interpreting the combined result becomes harder.
Long-running tests can also create operational problems.
Your team may postpone other improvements because the page is “currently under test.” A variant created months earlier may no longer reflect your current strategy.
For low-traffic websites, long tests are sometimes unavoidable. Our guide to A/B testing for low-traffic websites explains why those sites should focus on larger, more meaningful changes rather than spending months testing tiny differences.
The important point is that duration itself is not the problem.
Changing conditions are.
Seasonality Can Distort Results
Seasonality deserves special attention.
A retailer running an experiment during Black Friday is studying behavior under very unusual purchasing conditions.
A travel site running a test during the summer holidays may see a different audience than it does in January.
A B2B company testing during the final week of December may observe behavior that does not represent a normal working month.
This does not mean experiments should stop during seasonal periods.
Sometimes the seasonal period is exactly what you want to optimize.
If your goal is to improve the Black Friday checkout, then Black Friday traffic is the correct audience.
The problem arises when you take a result from an unusual period and assume it applies universally.
Always interpret the experiment in the context in which the traffic was collected.
Avoid Editing a Test While It Is Running
Another factor affecting duration is experiment consistency.
Once an A/B test begins, avoid changing the variants or conversion definitions unless there is a serious technical problem.
Imagine that version B uses one headline for the first week and a different headline for the second.
What exactly does B’s final conversion rate represent?
It combines results from two different experiences.
The same problem occurs if you change the conversion goal, targeting conditions, or important page elements halfway through the experiment.
Nelio A/B Testing prevents editing while a test is active. You can pause a test and make changes, but depending on the modification, the partial results may no longer be valid; in such cases, starting a new test is preferable.
If the experiment needs a major change, it is usually cleaner to stop it and begin a new one.
A Practical Example
Imagine a WordPress pricing page receiving 10,000 visits per month.
The original page converts 4% of visitors into trial registrations.
You create an alternative that simplifies the pricing table and emphasizes one recommended plan.
After three days, version B has a conversion rate of 5.6% while A has 3.9%.
The alternative looks excellent.
But the sample is still small and most of the traffic arrived from a newsletter sent on the day the experiment launched.
You keep the test running.
At the end of the first full week, A converts at 4.1% and B at 4.8%. The advantage still exists, but it is much smaller.
During the second week, both versions receive normal organic and paid traffic. The confidence level continues increasing.
By the end of the second week, B converts at 4.9% and A at 4.0%, with enough page views and conversions to produce a clear, high-confidence winner.
Stopping after three days would have selected the same version, but for the wrong reason.
The later result is more trustworthy because it includes a larger sample and a more representative traffic mix.
Now imagine that B instead falls to 4.1%.
The early apparent 40% improvement would have disappeared entirely.
This is exactly why patience matters.
Another Example: Low-Traffic Lead Generation
Consider a B2B consultancy whose service page receives 1,000 visits per month and produces around 30 form submissions.
The team tests a new value proposition.
After two weeks, the original has seven conversions and the alternative has nine.
There is not enough evidence to confidently distinguish the two.
After one month, A has 14 conversions and B has 17.
The alternative still looks somewhat better, but confidence remains low.
At this point, the team has several options.
If the change is strategically important and conditions remain stable, the experiment can continue.
If the observed difference is too small to have meaningful business value, the team may decide to stop and use the traffic for a stronger experiment.
There is no benefit in waiting three more months merely to obtain a statistically significant answer to an unimportant question.
This is where test duration becomes a business decision as well as a statistical one.
How Nelio A/B Testing Helps Decide When to Stop
Nelio A/B Testing is designed to make the state of an experiment visible without requiring users to calculate significance manually.
While an experiment is running, its results page shows visits, conversions, conversion rates, improvement over the control, and confidence information. As more data accumulates, the experiment can move from simply collecting data to showing a potential winner with low confidence and eventually a clear winner.
By default, Nelio recommends a minimum sample size of 500 page views and a required confidence of 95%, although both values can be adjusted in the settings.
Depending on the plan, you can also define automatic stopping rules based on duration, total page views, or confidence.
This means you do not have to choose between “always run for 14 days” and manually watching the dashboard every hour.
The experiment itself can tell you when enough evidence has accumulated.
A Better Way to Think About Test Duration
Instead of asking how many days an experiment should run, think about four conditions.
First, has the experiment collected enough traffic to avoid making a decision from a tiny sample?
Second, have enough conversions occurred to compare the variants meaningfully?
Third, has the result reached a confidence level appropriate for the importance of the decision?
Finally, has the experiment covered a representative period of user behavior rather than an unusual traffic spike or incomplete business cycle?
If those conditions are satisfied, continuing simply because you originally wrote “30 days” in a planning document adds little value.
If they are not satisfied, stopping because “two weeks have passed” is equally arbitrary.
Frequently Asked Questions
How many days should an A/B test run?
There is no universal number.
The correct duration depends on traffic, conversions, the difference between variants, statistical confidence, and the normal behavior cycle of your audience.
Whenever possible, allow the test to cover at least one representative business cycle and wait until enough evidence has accumulated.
Is two weeks enough for an A/B test?
It can be, but not always.
Two weeks may be more than enough for a high-traffic experiment with frequent conversions and a large effect. It may be far too short for a low-traffic page with a rare conversion.
Use the accumulated data rather than the calendar alone.
Can an A/B test run too long?
Yes, if continuing no longer provides useful information or if the conditions surrounding the experiment change significantly.
A long-running test may also prevent you from testing more valuable ideas.
If the variants remain almost identical after substantial traffic, accepting that there is no meaningful difference can be more useful than waiting indefinitely for a winner.
What confidence level should I use?
95% is a common benchmark and the recommended default in Nelio A/B Testing.
Lower thresholds allow faster decisions with more uncertainty. Higher thresholds require more evidence.
The appropriate value depends partly on how costly an incorrect decision would be.
How many visitors does an A/B test need?
There is no single number that works for every experiment.
Nelio uses a configurable minimum sample size and recommends at least 500 page views before declaring a winner, but many tests require significantly more data.
Conversion frequency and the size of the difference between variants are critical.
Should I stop a test when statistical significance is reached?
Usually, reaching your confidence threshold is an important signal, but also consider whether the traffic is representative.
If significance is reached during an unusual campaign or after only one narrow part of the weekly traffic cycle, allowing the experiment to continue may provide a more robust result.
What if my test has a winner after one day?
Be cautious.
Early results can fluctuate dramatically because the sample is small.
Even if the confidence appears promising, make sure the experiment has sufficient traffic, conversions, and representative coverage before making an important decision.
Should I stop an A/B test if there is no winner?
Eventually, yes.
Not every experiment needs to produce a winner.
If enough traffic has accumulated and the difference remains commercially insignificant, the correct learning may be that the change does not matter enough to pursue.
Can I automatically stop an A/B test?
Yes.
Nelio A/B Testing supports several end modes depending on your plan, including stopping based on elapsed duration, total page views, or reaching the configured confidence level.
Final Thoughts
The best A/B tests do not end because they have been running for exactly 7, 14, or 30 days.
They end because enough evidence has accumulated to support a decision.
Traffic determines how quickly visitors enter the experiment. Conversion frequency determines how quickly meaningful events accumulate. The size of the difference between variants determines how easily that difference can be detected.
Statistical confidence helps determine whether the observed result is likely to be real.
But statistics are only part of the decision.
The experiment should also cover a representative period of visitor behavior, avoid unusual distortions, and remain valuable enough to justify the traffic it consumes.
For some websites, that may mean one week.
For others, it may mean six weeks.
And sometimes the right decision is to stop an inconclusive experiment and move on to a stronger hypothesis.
Nelio A/B Testing tracks visits, conversions, conversion rates, improvement, and statistical confidence so WordPress users can monitor how evidence develops over time rather than relying on an arbitrary calendar rule.
It can also stop experiments automatically according to predefined conditions and apply the winning WordPress variant once the test is complete.
If limited traffic is the reason your experiments take longer, read our guide to A/B testing for low-traffic websites.
And if you are still looking for a testing solution, see our comparison of the best A/B testing plugins for WordPress.
Download Nelio A/B Testing for free or compare the available plans to run your first experiment.
Do not ask whether your test has been running long enough.
Ask whether you have learned enough to make the decision.
Leave a Reply