A/B Testing On Click Through Rates

Description

Suppose for example that on the front page of our website we have a "Buy" button, where viewers can pay for a trial of the software. But the button is beneath some explanatory content, and we are worried that users aren't seeing it, since in a focus group we saw that some users didn't scroll down the page. We want to design an A/B test to see if moving the button to the top of the site improves the click-through rates of our trial. More specifically, we construct the following hypothesis: 

Because we noticed that not all users scroll to the bottom of the page we expect that moving the "Buy" button to the top of the page will lead more users to purchase. We will measure this by showing the new page to a percentage of viewers and recording whether or not they click the "Buy" button. 

ETL

In this use case, the results of the two pages are stored as two different CSV files. Each row includes a visit to the page and whether or not the user clicked the Buy button. Because we want to compare the data in these two datasets, we need to combine them into one long dataset. We can combine the two operators using the Set Operations operator. However, because we need to remember which dataset each row came from, we will first need to use a Variable operator to create a new column with the version of the website that the user saw. 

Here is a screenshot of this ETL operation. 

The resulting data looks like this: 

After creating this new dataset, we can do our analysis. 

We want to see if the version that the users saw had a significant effect on the number of clicks to sign up. We can test this using a simple Chi-Square Test of Independence. Our null hypothesis is that there is no statistical relationship between users who saw the new web page and users that clicked to sign up for the free trial. Thus we select our dependent variable as "Clicked_To_Buy". Our independent variable is "Version" ? the version of the website that the user saw: 

Our results show that the test was not significant, so we cannot reject the null hypothesis.