# Bayesian A/B testing in R

# Introduction to bayesian A/B testing

## Part 1

Watch this amazing series if you need an introduction to baysian.

## Part 2

## Part 3

## Loading packages

Install bayesAB if you do not have it.

```
# Loading packages
library(tidyverse)
library(bayesAB)
```

## Loading data

```
# Loading data
set.seed(20220513)
=rbinom(n=16,size=1,prob=0.8)
Atable(A)
```

```
A
0 1
3 13
```

```
=rbinom(n=100,size=1,prob=0.6)
Btable(B)
```

```
B
0 1
42 58
```

# Checking the maximum likelihood

`mean(A)`

`[1] 0.8125`

`mean(B)`

`[1] 0.58`

## We have no idea what lies in the data

Using a uninformative prior.

## Setup for the test

Using Bernoulli distribution with 100k samples.

```
<- bayesTest(A, B, priors = c('alpha' = 1, 'beta' = 1),
AB1 n_samples = 1e5, distribution = 'bernoulli')
print(AB1)
```

```
--------------------------------------------
Distribution used: bernoulli
--------------------------------------------
Using data with the following properties:
A B
Min. 0.0000 0.00
1st Qu. 1.0000 0.00
Median 1.0000 1.00
Mean 0.8125 0.58
3rd Qu. 1.0000 1.00
Max. 1.0000 1.00
--------------------------------------------
Conjugate Prior Distribution: Beta
Conjugate Prior Parameters:
$alpha
[1] 1
$beta
[1] 1
--------------------------------------------
Calculated posteriors for the following parameters:
Probability
--------------------------------------------
Monte Carlo samples generated per posterior:
[1] 1e+05
```

### Checking out the results from the summary

`summary(AB1)`

```
Quantiles of posteriors for A and B:
$Probability
$Probability$A
0% 25% 50% 75% 100%
0.2445932 0.7171955 0.7879369 0.8483182 0.9912624
$Probability$B
0% 25% 50% 75% 100%
0.3743089 0.5462008 0.5793034 0.6119321 0.7599387
--------------------------------------------
P(A > B) by (0)%:
$Probability
[1] 0.95664
--------------------------------------------
Credible Interval on (A - B) / B for interval length(s) (0.9) :
$Probability
5% 95%
0.01391909 0.68653184
--------------------------------------------
Posterior Expected Loss for choosing A over B:
$Probability
[1] 0.004511233
```

### Plotting the results

`plot(AB1)[[2]]`

`$Probability`

`$Probability`

Posterior Expected Loss for choosing A over B: 4.5 percent.

# Rasmus Bååth example - Using one distribution

I run through the examples from Rasmus Bååth. Check it out.

We will send 16 mails to people and 6 signed up, how good is this method? We are searching for the probability that a random person who receives the offer will signup. We know that is possible to extract the maximum likelihood.

\[\frac{Signed\ up}{Recieved\ Offer}=\frac{6}{16}=0.375=37.5\ percent\]

16 however is a small sample of people, how certain can we be that it is giving us any value. We can use bayesian methods to extract the possible ways to reach 16 people.

### What would the rate of sign-up be if method A was used on a larger number of people?

```
# Number of random draws from the prior
<- 10000
n_draws
<- runif(n_draws,0,1) # Here you sample n_draws draws from the prior
prior hist(prior) # It's always good to eyeball the prior to make sure it looks ok.
```

```
# Here you define the generative model
<- function(rate) {
generative_model =rbinom(1,size=16,prob=rate)
subs
subs
}
# Here you simulate data using the parameters from the prior and the
# generative model
<- rep(NA, n_draws)
subs for(i in 1:n_draws) {
<- generative_model(prior[i])
subs[i]
}
# Here you filter off all draws that do not match the data.
<- prior[subs == 6]
posterior
hist(posterior) # Eyeball the posterior
```

`length(posterior) # See that we got enought draws left after the filtering.`

`[1] 605`

```
# There are no rules here, but you probably want to aim
# for >1000 draws.
# Now you can summarize the posterior, where a common summary is to take the mean
# or the median posterior, and perhaps a 95% quantile interval.
median(posterior)
```

`[1] 0.3837504`

`quantile(posterior, c(0.025, 0.975))`

```
2.5% 97.5%
0.1805234 0.6013402
```

`sum(posterior > 0.2) / length(posterior)`

`[1] 0.9636364`

```
2.5% 97.5%
17 63
```

If we send the mail to 100 people it is likely we will see it being between 16 and 61 people.

# End

Thanks for reading this post, this has been a post on A/B testing in R using Bayesian methods. It is a superb way for data scientist to discover what version is the best. Ciao!