Experimentation and A/B testing
A test that is set up badly gives an answer that feels reliable and isn't. We help you design experiments properly, run them for the right length of time and read the results honestly.
Signs you could use this
- Tests are stopped as soon as the numbers look good.
- Results disagree from one week to the next.
- You're not sure how long to run a test or how many users you need.
- Wins in testing don't show up in the overall numbers.
What we do
- Design the test
- A clear hypothesis, one primary metric, guard-rail metrics and a sample size worked out in advance, not after the fact.
- Check the set-up
- Random assignment, tracking and data quality are verified before launch. Many failed tests fail here.
- Analyse carefully
- Appropriate statistics, treatment of outliers and honest handling of segments. We report uncertainty as well as the headline difference.
- Build the habit
- A simple experiment log and review routine, so learning builds up and tests are not repeated by accident.
What you receive
Short advisory engagements of two to four weeks work well. Some teams keep us on a monthly basis to review each test.
- A test design template and sample size calculator for your traffic
- Analysis of your current and past tests
- An experiment log and review process
- Training for product and marketing teams
Tools we use: SQL, Python, R, Mixpanel, Google Analytics, Your existing testing platform.
Common questions
We don't have much traffic. Can we still test?
Sometimes. For small volumes we look at bigger changes, longer tests or other evidence such as before and after comparisons. We'll be honest when a test isn't worth running.
Which testing tool should we use?
Whichever you already have is usually fine to start with. The method matters far more than the platform.
Can you re-analyse tests we've already run?
Yes. Reviewing old tests often shows which past decisions were solid and which were luck.