Part IV - Terms - A/B Testing

Question 1

When is A/B testing a good idea?

Answer 1

When you have clear measurable goals and are testing specific theories

Question 2

When is A/B testing a bad idea?

Answer 2

When goals are vague and designs have multiple variables

Question 3

How many variations should be in A/B testing?

Answer 3

Two at a time to maintain clarity in results

Question 4

What is a null hypothesis in A/B testing?

Answer 4

The assumption that any difference in outcomes is due to chance

Question 5

How many visits are needed for significant A/B test results?

Answer 5

Enough to reach a statistical significance of 95% confidence

Question 6

What is multivariate testing?

Answer 6

Testing multiple variables to see how they interact over time

Question 7

Does Google think negatively about A/B testing?

Answer 7

No; Google supports A/B testing as it does not count as duplicate content

Question 8

How and when to interpret A/B testing results?

Answer 8

After reaching statistical significance to accurately reflect on the hypothesis tested

Question 9

How many variables should be tested in A/B testing?

Answer 9

One at a time for clear actionable results

Question 10

What should be tested in A/B testing?

Answer 10

Elements like CTAs, headlines, images, and copy length

Question 11

Can A/B tests be run on non-webpage elements?

Answer 11

Yes, including emails, PPC campaigns, and CTAs

Question 12

How to find A/B tests from similar companies?

Answer 12

Check websites like “Which Test Won” and “Visual Website Optimizer” for examples

Question 13

What if A/B test results are not trusted?

Answer 13

Re-run the test to verify the results

Question 14

How often should A/B testing be conducted?

Answer 14

Continuously as long as each test has a clear purpose

Question 15

What is needed to start A/B testing on a site?

Answer 15

A/B testing software like HubSpot or Google Content Experiments

Question 16

What are some validity traps/effects in A/B testing?

Answer 16

History effect and Instrumental effect, among others

Question 17

Should I A/B test my homepage?

Answer 17

Yes, but consider testing specific elements like CTAs due to variable visitor goals

Question 18

What if there is no control in an A/B test?

Answer 18

Make one; designate one variation as the control (typically the one most similar to current design)

Question 19

Why isn’t A/B testing split exactly 50/50?

Answer 19

Due to random variations (but it balances out with more traffic)