Interpret A/B test results with sanity checks

Paste your experiment numbers and get a plain-language verdict, validity checks, and a clear ship, wait or stop recommendation.

LLazyPrompt(@lazyprompt)Data & AnalysisAnalyzeWorks with any model
Prompt template
Act as a careful experimentation analyst. Interpret these A/B test results and tell me what decision they support.

Experiment: {{experiment}}
Primary metric: {{metric}}
Control: {{control_data}}
Variant: {{variant_data}}
Planned duration and sample size, if any: {{plan}}
Other context (secondary metrics, segments, known issues): {{context}}

Work through these steps and show the arithmetic:
1. Effect: absolute and relative difference, with a 95% confidence interval. State which test you used and why. If I did not give enough numbers to compute it (sample sizes, conversions or means and standard deviations), say what you need.
2. Sample ratio mismatch: check whether the traffic split matches the intended split. If it does not, flag that the result is unreliable.
3. Validity checks: whether the test ran long enough (full weekly cycles), signs of peeking or stopping early, novelty effects, multiple comparisons if several metrics or segments were examined, and any instrumentation concerns.
4. Practical significance: translate the interval into business terms (for example extra orders per month) using the figures I gave, and say whether the whole interval is worth acting on.
5. Verdict: ship, keep running, or stop, with the main reason in one sentence and the main risk in one sentence.
6. Explain the result for a non-technical stakeholder in 3 sentences, without the word "significant" unless defined.

Do not call a result a win just because the p-value is below 0.05. State plainly when the data cannot support a decision.
Example output
**1. Effect** Control 7.50% (3,090/41,200), variant 8.01% (3,230/40,300). Absolute lift +0.51 pts, relative +6.9%. Two-proportion z-test: z = 2.69, p = 0.007. 95% CI for the difference: +0.14 to +0.88 pts. **2. Sample ratio mismatch** Intended 50/50; observed 41,200 vs 40,300 (50.6% / 49.4%). Chi-square p = 0.04. This is a mild imbalance that is borderline; investigate before trusting the result (bot filtering, redirects, or a slower variant page losing sessions). **3. Validity** - 14 days = two full weekly cycles: good. - No segment analysis, so no multiple-comparison problem yet. Do not add one after the fact. - Novelty effect possible for a colour change; check lift by week. **4. Practical significance** At 40,000 mobile sessions per week, +0.14 to +0.88 pts is roughly 56 to 350 extra orders a week. **5. Verdict**: Keep running, or rerun, until the sample ratio mismatch is explained. Main reason: the imbalance could bias the lift. Main risk: shipping a change whose effect comes from a tracking artifact. **6. Plain-English summary** The green button looks like it converts a bit better, probably between a small and a decent amount. But the two groups were not split evenly, which can distort results, so we should check that first.

Why it works

Mandatory sample ratio and validity checks stop the common error of declaring wins from p-values alone. Include weekly breakdowns if you have them for a novelty-effect check.

Licensed CC0 (public domain) Version 1 · history