Methodology: Frequentist and Bayesian are the two main types of A/B testing methods. Both methods allow you to compare different versions of content to see which one is better. They differ in their approach to calculating and interpreting results. Candu uses the Bayesian method.

How it works: Users are randomly split into different groups for each content or control group version. We do not have prior knowledge or historical data to use, so we use an uninformed prior (default value selected) - essentially a starting point of "I don't know" how this will likely perform.

What it tells us: With the Bayesian method of A/B testing, instead of saying, "Yes, this version is better" or "No, this version is worse," results are expressed as a probability distribution. We use probabilities to measure the likelihood that one version is better. For example, you can say, "There is an 85% probability that Version A is better than the control group, but we can't be 100% sure."

Small sample sizes: Bayesian methods can be more reliable when you have relatively small sample sizes (not many users to test on). In some cases, Frequentist methods struggle to provide confident results with limited data. Bayesian methods handle small sample sizes better and give you more reliable insights. For in-product experiments and B2B businesses, having reliable results with small sample sizes is crucial.

No peeking problem: With the Bayesian method, results are valid at any point and will not be skewed by unusually large results. For example, if most upgrades are for $10, but a single upgrade is for $10,000, this will not unduly skew the experiment results. (The peeking problem can occur with the Frequentist approach when you check the intermediate results for statistical significance between the control and test groups and make decisions based on your observations.)

Probabilities improve over time - As new information comes in, your probabilities will continue to be refined. This means you can keep learning and adapting your decisions throughout your experiment.

Results interpretation: Probability-based results from Bayesian testing are often easier to understand and explain to non-technical stakeholders. Rather than communicate results with p-values and statistical significance, you can say, "There's a high chance that the new design will attract more users."

Informative results: Because Bayesian A/B testing provides probabilistic statements instead of just a simple 'yes' or 'no,' this means you better understand how likely one version is that it is better than the other. Saying, "There's a 90% chance that Version A is better than Version B," gives you a more nuanced and informative answer.

Rows: Each of the goals you have set for your experiment will appear as a new row, allowing you to measure the conversion rate and impact for each.

Columns: Each version/variant and/or control group will appear in a new column, allowing you to compare results across all options.

Users: This is the total number of your users who have viewed each version.

Count: the total number of event counts/total users who did the action

Conversion: the total users who converted/ total users who viewed the content

Revenue: the total revenue generated/total users who contributed

Conversion rates for each version

Total content views and unique content views per version

Any interactions that have occurred in your content