How Does Nelio A/B Testing Calculate Confidence and Statistical Significance?

Nelio A/B Testing uses the G-test (likelihood-ratio test) to determine whether the differences observed between variants are statistically significant.

The G-test compares the results collected during your test with the results that would be expected if there were no meaningful difference between the variants. This helps determine whether the observed differences are likely to reflect a real effect rather than random variation.

Nelio A/B Testing performs these calculations automatically and displays the resulting confidence level in your test results.

What Does the Confidence Level Mean?

The confidence level indicates the strength of the statistical evidence that one variant performs better than the others.

A higher confidence level means there is stronger evidence that the observed difference between variants is not simply due to random variation. A lower confidence level means that more data may be needed before drawing a reliable conclusion.

By default, Nelio A/B Testing considers a winning variant to be statistically significant when the confidence level reaches 95%.

Why Does the Confidence Level Change Over Time?

The confidence level is recalculated as your test collects more data, so it may increase or decrease while the test is running.

It is affected by factors such as:

• the number of visitors included in the test;
• the number of conversions generated by each variant; and
• the difference in performance between variants.

If the variants perform similarly or the test has collected relatively little data, the confidence level may remain low. As more data is collected and a consistent difference between variants emerges, the statistical evidence may become stronger.

🔍 Learn more in How Does Nelio A/B Testing Decide on a Winning Variant?