A test brief the team can evaluate without changing the rules afterward.
Write a falsifiable hypothesis
Describe the customer problem, proposed change and expected outcome. Explain what evidence would make you reconsider. A hypothesis should be specific enough that a result can challenge it.
Choose a primary outcome and a small number of guardrails. For example, a change intended to improve conversion should also be checked for effects on order economics or customer experience.
Choose a suitable comparison
A before-and-after comparison is vulnerable to seasonality, promotions and changes in traffic. Use a controlled test when you need to isolate an effect, and check whether the available traffic supports the question.
- Define the eligible audience and assignment method.
- Record the planned duration and sample considerations.
- Check the measurement and investigate overlapping changes.
- Agree on stopping conditions before the test begins.
Keep the uncertainty in the result
Review the outcome using the planned method. Do not repeatedly stop a test as soon as a favorable number appears. If the sample is insufficient or conditions changed, record the result as inconclusive.
Store the hypothesis, execution details and conclusion together. The next team member should be able to understand what the test actually established.
Your next steps
Use this checklist as you work through the guide.