Question 1

What is a p-value in A/B testing?

Accepted Answer

A p-value is the probability of observing results at least as extreme as the measured conversion lift, assuming the null hypothesis of no real difference is true. A p-value below 0.05 indicates statistical significance at 95% confidence.

Question 2

What is the danger of peeking early during an A/B test?

Accepted Answer

Repeatedly checking p-values and stopping the test as soon as p < 0.05 drastically inflates false positive (Type I error) rates from the nominal 5% to over 30%, leading to false winners.

Question 3

What is the difference between relative lift and absolute difference?

Accepted Answer

Absolute difference is the direct percentage point subtraction (5% - 4% = 1% difference), while relative lift expresses that change proportional to the baseline ((5% - 4%) / 4% = 25% relative uplift).

A/B Test Significance Calculator

Frequently Asked Questions

Math · Related Calculators & Tools