A/B Testing
There could be different opinions about choices. Have you ever seen survey’s in social medias such as “which color of button in UI looks good?”, “What shape of button is good?”, “What text to keep on call on action button?”
To resolve these problems, we have A/B testing. It gives best choice, which performs well amongst different versions. A base design is called control. Newly created design — variation is built on top of control, is a different version.
Divide target group between Control(A) and Variation(B). Measure performance of both the designs, Evaluate the results with the metrics which are important for the business.
Steps of Hypothesis Testing:
- Set up the null and alternative hypotheses, and decide the significance level.
- Calculate the sample statistic.
- Formulate the test statistic.
- Make the decision criteria.
- Calculate the test statistic.
- Take decision.
Consider there are two population proportion where we want to test if proportion of desired characteristic is statistically same in both populations.
Null hypothesis would be : Proportion of population1 and population2 are equal
Considering significance level 0.05, The value of z is 2.1343. The value of p is .03318. The result is significant at p < 0.05, critical z-values are -1.96 and +1.96 for two tailed test. Reject null hypothesis as 2.13 is not in acceptance region.
A/B Testing Steps:
Analysis of data
Qualitative data and Quantitative data answer what to test and how to test. We have data like-
- Transaction history data
- Traffic data from web
- Population test
- Customer support requests
Hypothesis formulating
This has a format — if (variable), then(result), because(reason)
Example, If we fill butterfly with color, then more number of people are clicking on it because this is an insight from the test conducted
Construct experiment
Increasing sample size would reduce Type1 and Type2 errors in Hypothesis testing.
A/B test parameters:
Duration of test, goal, sample size are also important parameters.
Interpreting results of test
Conclude statistically A or B, which one is performing better.
A/B test is completely data driven test (not hunch driven).
Thank you for reading :)
