Interpret A/B test results with sanity checks

Paste your experiment numbers and get a plain-language verdict, validity checks, and a clear ship, wait or stop recommendation.

Change note: Initial version

Prompt

Act as a careful experimentation analyst. Interpret these A/B test results and tell me what decision they support.

Experiment: {{experiment}}
Primary metric: {{metric}}
Control: {{control_data}}
Variant: {{variant_data}}
Planned duration and sample size, if any: {{plan}}
Other context (secondary metrics, segments, known issues): {{context}}

Work through these steps and show the arithmetic:
1. Effect: absolute and relative difference, with a 95% confidence interval. State which test you used and why. If I did not give enough numbers to compute it (sample sizes, conversions or means and standard deviations), say what you need.
2. Sample ratio mismatch: check whether the traffic split matches the intended split. If it does not, flag that the result is unreliable.
3. Validity checks: whether the test ran long enough (full weekly cycles), signs of peeking or stopping early, novelty effects, multiple comparisons if several metrics or segments were examined, and any instrumentation concerns.
4. Practical significance: translate the interval into business terms (for example extra orders per month) using the figures I gave, and say whether the whole interval is worth acting on.
5. Verdict: ship, keep running, or stop, with the main reason in one sentence and the main risk in one sentence.
6. Explain the result for a non-technical stakeholder in 3 sentences, without the word "significant" unless defined.

Do not call a result a win just because the p-value is below 0.05. State plainly when the data cannot support a decision.

Variables

• {{experiment}}Experimenttext

  Default: Green vs blue checkout button on the mobile checkout page

• {{metric}}Primary metrictext

  Default: Checkout conversion rate (orders / checkout sessions)

• {{control_data}}Control resultstext

  Default: 41,200 sessions, 3,090 orders

  Visitors and conversions, or mean and standard deviation.

• {{variant_data}}Variant resultstext

  Default: 40,300 sessions, 3,230 orders

• {{plan}}Plantext

  Default: Intended 50/50 split, 14 days, about 40,000 sessions per arm

• {{context}}Extra contextlong

  Default: Ran Mon 3 Mar to Sun 16 Mar. Average order value was unchanged. No segment analysis done.

  Secondary metrics, segments, incidents.

Example output

**1. Effect**
Control 7.50% (3,090/41,200), variant 8.01% (3,230/40,300). Absolute lift +0.51 pts, relative +6.9%. Two-proportion z-test: z = 2.69, p = 0.007. 95% CI for the difference: +0.14 to +0.88 pts.

**2. Sample ratio mismatch**
Intended 50/50; observed 41,200 vs 40,300 (50.6% / 49.4%). Chi-square p = 0.04. This is a mild imbalance that is borderline; investigate before trusting the result (bot filtering, redirects, or a slower variant page losing sessions).

**3. Validity**
- 14 days = two full weekly cycles: good.
- No segment analysis, so no multiple-comparison problem yet. Do not add one after the fact.
- Novelty effect possible for a colour change; check lift by week.

**4. Practical significance**
At 40,000 mobile sessions per week, +0.14 to +0.88 pts is roughly 56 to 350 extra orders a week.

**5. Verdict**: Keep running, or rerun, until the sample ratio mismatch is explained. Main reason: the imbalance could bias the lift. Main risk: shipping a change whose effect comes from a tracking artifact.

**6. Plain-English summary**
The green button looks like it converts a bit better, probably between a small and a decent amount. But the two groups were not split evenly, which can distort results, so we should check that first.

Notes

Mandatory sample ratio and validity checks stop the common error of declaring wins from p-values alone. Include weekly breakdowns if you have them for a novelty-effect check.