Hypothesis Testing Foundations for A/B Testing
Let's go into A/B Testing directly with an example. An e-commerce website wants to increase the conversion rate for its product. One potential improvement that could be made is the product detail page layout.
Version A might be quite basic, containing elements such as product pictures, descriptions, prices, and an 'add to cart' button.
In contrast, Version B could include enhancements like user comments, adjustments to the size and color of the 'add to cart' button, and a promo code.
When the experiment starts, half of the website's traffic will be directed to Version A, and the other half to Version B. This process may be run anywhere between weeks and months. All user logs and activities are recorded. Analysts calculate the conversion rate of each group. If Version B shows a higher conversion rate than Version A with statistical significance, this confirms the efficacy of Version B and will lead to the decision to upgrade the version.
This is a simple classical example of how the A/B test is adopted in companies. The testing components can be metrics, like click-through rate, conversion rate, ROI, etc; UI designs like button size and the famous 41 Shades of Blues experiment by Google for logo color; or app features.
Good times to use A/B test:
Unnoticeable changes like loading time
Change in UI
Launch a small feature
Something that can be tested in a short timeframe
Bad time to use A/B testing:
Totally new experience: impact of upgrading brand and theme
Unethical to test: abusive content, scams, fake users
Components that cannot be controlled: customer service
Metrics that take longer time to track: retention rate, referral effect
In essence, A/B testing is an applied case of hypothesis testing. The null hypothesis is two versions are the same, and the alternative hypothesis is one is better than the other. With randomization, data collection, and statistical analysis, we calculate p-values and conclude by accepting or rejecting the null hypothesis with a certain confidence level.
What is the p-value?
The p-value is a key concept in hypothesis testing. Throughout these tests, we calculate the p-value and compare it to a predetermined threshold. The p-value represents the probability of observing the data purely by chance. For example, to determine whether group A has a higher mean than group B, a calculated p-value of 0.03 means there is a 3% chance that the observed difference in means occurred randomly, purely by luck (not a consistent observable phenomenon). Since it’s lower than 5%, the maximum probability we can accept, we will say this observation is 97% likely to be consistent. Subsequently, we can conclude group A indeed has a higher mean than group B with 97% confidence.
Graphically, the p-value can be represented as the shaded area corresponding to the test statistic. The point on the distribution curve indicates the position of our observed data among the entire distribution under the null hypothesis. If it lies far from the center, it suggests a low probability of the result occurring under the null hypothesis, which leads to rejection of the null hypothesis, acceptance of the alternative hypothesis, and being statistically significant.
There are hundreds of different hypothesis tests available. It’s important for us to understand when is the optimal scenario to use each test.
Z Test
The Z-test is used when we want to compare the mean between only two groups when the population parameters like mean and variance are known, which is very rare in practice.
Steps:
Null hypothesis (H0): two groups have the same mean. Alternative hypothesis (H1): two groups have different means (two-tailed test), or group A has a larger mean than group B (one-tailed test).
Calculate the Z-statistic:
Convert Z statistics to p-value and compare to the selected confidence threshold
If p-value < alpha, reject the null hypothesis with (1-alpha) confidence, otherwise accept it.
import numpy as np
from statsmodels.stats.weightstats import ztest
# Generate two sets of random data for two groups
group1 = np.random.normal(100, 10, size=50)
group2 = np.random.normal(105, 10, size=50)
# Perform the Z-test to compare the two groups
z_statistic, p_value = ztest(group1, group2)
print(f'Z-statistic: {z_statistic}, P-value: {p_value}')
# Z-statistic: -2.022844383911619, P-value: 0.04308919406283927With p-value < 0.05, we conclude that we are 95% confident that the two groups have different means, which is true in this case as we know the true data generation process. Yet in general, we do not know, the confidence level in the conclusion should never be ignored.
Paired Sample T-Test
The Paired Sample t-test is also known as the Dependent Sample t-test. It is used when we compare two group means with unknown population parameters. And the inter-individual variability is large within the group. It is commonly used in medical fields rather than products, to test the effectiveness of a treatment among patients. The prior and post-treatment of the same group are the two groups for this test.
import numpy as np
from statsmodels.stats.weightstats import ttest_ind
# Example data: two related samples
before = np.random.normal(100, 10, size=30)
after = before + np.random.normal(5, 2, size=30) # e.g. effect of a treatment
# Perform the paired t-test
t_statistic, p_value, _ = ttest_ind(before, after, usevar='pooled', alternative='two-sided', value=0)
print(f'T-statistic: {t_statistic}, P-value: {p_value}')
# T-statistic: -2.0037435380355904, P-value: 0.04977611134259076Independent Sample T-Test
Defined as opposed to a Paired sample t-test, the Independent sample t-test is used to compare the mean of two independent groups with unknown population parameters. The test further incorporates an assumption of equal or unequal variances between the two groups.
The assumption of equal variances is known as the homoscedasticity assumption.
import numpy as np
from statsmodels.stats.weightstats import ttest_ind
# Example data: two independent samples with equal variances
group1 = np.random.normal(100, 10, size=30)
group2 = np.random.normal(110, 10, size=30)
# Perform the independent t-test
t_statistic, p_value, _ = ttest_ind(group1, group2, usevar='equal', alternative='two-sided', value=0)
print(f'T-statistic: {t_statistic}, P-value: {p_value}')
# T-statistic: -3.7264113962728502, P-value: 0.00044478800824450496Welch T-Test
The Welch t-test is a variant of the Independent t-test when variances of two groups are assumed to be unequal.
import numpy as np
from statsmodels.stats.weightstats import ttest_ind
# Example data: two independent samples with unequal variances
group1 = np.random.normal(100, 10, size=30)
group2 = np.random.normal(110, 15, size=40)
# Perform the Welch t-test
t_statistic, p_value, _ = ttest_ind(group1, group2, usevar='unequal')
print(f'Welch t-statistic: {t_statistic}, P-value: {p_value}')
# Welch t-statistic: -3.740779777950355, P-value: 0.0003954108847525618ANOVA
All the above approaches are designed for cases with only two groups. When there are three or more groups, we use ANOVA (Analysis of Variance).
Steps:
Background: we are comparing three or more group means, assuming equal variances and independency among all groups.
The null hypothesis (H0) states that all group means are equal, and the alternative hypothesis (H1) states that at least one group mean is different.
Calculate F-statistic
Compare and conclude
import numpy as np
import statsmodels.api as sm
from statsmodels.formula.api import ols
# Example data
data = {'Score': np.concatenate([np.random.normal(100, 10, size=30),
np.random.normal(105, 10, size=30),
np.random.normal(110, 10, size=30)]),
'Group': ['A']*30 + ['B']*30 + ['C']*30}
# Perform ANOVA
model = ols('Score ~ C(Group)', data=data).fit()
anova_results = sm.stats.anova_lm(model, typ=2)
print(anova_results)
Pr(>F) is the p-value corresponding to F-statistics, we compare it with an alpha of 0.05 and conclude that there is at least one group with a different mean at 95% confidence.
Chi-Squared Test
When the dataset is numerical, we can compare the sample mean. But when data is categorical (e.g. gender, color, brand), we are unable to calculate a mean. In this case, we introduce the Chi-squared test.
Steps:
Create a contingency table by summarizing the frequencies of variables.
Calculate the Chi-squared statistic
import numpy as np
from scipy.stats import chi2_contingency
# Example data: Contingency table in a 2x2 format
contingency_table = np.array([[30, 70], [45, 55]])
# Perform the Chi-squared test
chi2_stat, p_value, df, expected = chi2_contingency(contingency_table)
print(f'Chi-squared statistic: {chi2_stat}, P-value: {p_value}, Degrees of freedom: {df}')
# Chi-squared statistic: 4.181333333333333, P-value: 0.04087153408900628, Degrees of freedom: 1Mann-Whitney U Test
There is another scenario when data is indeed numerical but it does not make sense to use mean. For example, in movie rating, users rate movies on a scale of 0 to 10. However, each individual's level of appreciation with the same score is different. A generous rater may give 8 for average movies while a picky rater gives 3. If we calculate the mean brutally, we are ignoring the value of individual nuances. Instead, we can compare the median of the two groups using the Mann-Whitney U Test.
from scipy.stats import mannwhitneyu
import numpy as np
# Example data
group1 = np.random.normal(100, 10, size=30)
group2 = np.random.normal(110, 10, size=30)
# Perform Mann-Whitney U test
u_statistic, p_value = mannwhitneyu(group1, group2)
print(f'Mann-Whitney U statistic: {u_statistic}, P-value: {p_value}')
# Mann-Whitney U statistic: 312.0, P-value: 0.04206682132383563Kolmogorov-Smirnov (KS) Test
Sometimes it’s even unreasonable to reduce data to the sample median. Alternatively, we can compare the entire distribution using the Kolmogorov-Smirnov (KS) Test.
from scipy.stats import ks_2samp
import numpy as np
# Example data
group1 = np.random.normal(100, 10, size=30)
group2 = np.random.normal(110, 10, size=30)
# Perform Kolmogorov-Smirnov test
ks_statistic, p_value = ks_2samp(group1, group2)
print(f'KS statistic: {ks_statistic}, P-value: {p_value}')
# KS statistic: 0.4666666666666667, P-value: 0.0025300622362698397Kruskal-Wallis Test
When we are comparing three or more group distributions, we use the Kruskal-Wallis Test.
from scipy.stats import kruskal
import numpy as np
# Example data: three independent samples
group1 = np.random.normal(100, 10, size=30)
group2 = np.random.normal(110, 10, size=30)
group3 = np.random.normal(120, 10, size=30)
# Perform Kruskal-Wallis test
h_statistic, p_value = kruskal(group1, group2, group3)
print(f'Kruskal-Wallis H statistic: {h_statistic}, P-value: {p_value}')
# Kruskal-Wallis H statistic: 40.87394383394383, P-value: 1.3314837474142688e-09Summary
In the diagram below, we have summarized a list of questions to consider when deciding which model to use. In practical A/B tests, the t-test is always the most commonly used one. The technical challenges often lie somewhere else, such as experiment design, how to conduct the randomization (by username, region, IP, etc), metric selection, and determining sample size to validate a minimum detectable effect.
We are only scratching the surface...
If you're interested in learning more about how to implement A/B testing to your use case, feel free to reach out to us by leaving your contact information!
View certificate