Back to prompts
Data & AnalysisPremiumadvanced
4.7

Interpret A/B Test Results Like a Data Scientist

Get statistical significance analysis, practical significance, and clear next steps from any A/B test

Copy & Paste this prompt
I ran an A/B test and need help interpreting the results.

Test details:
- What I tested: [DESCRIBE THE CHANGE — e.g., new button color, different headline]
- Metric measured: [PRIMARY METRIC — e.g., conversion rate, click-through rate]
- Test duration: [HOW LONG IT RAN]

Results:
- Control (A): [SAMPLE SIZE] visitors, [CONVERSIONS] conversions ([RATE]%)
- Variant (B): [SAMPLE SIZE] visitors, [CONVERSIONS] conversions ([RATE]%)
- Any secondary metrics: [LIST THEM]

Analyze this test:

1. STATISTICAL SIGNIFICANCE
   - Calculate the p-value and confidence interval
   - Is this result statistically significant at 95% confidence?
   - Was the sample size sufficient? What would be needed?

2. PRACTICAL SIGNIFICANCE
   - What is the absolute lift and relative lift?
   - Is this difference meaningful in business terms?
   - Calculate the projected annual impact (if I give you revenue/user data)

3. VALIDITY CHECK
   - Was the test duration long enough? (full business cycles)
   - Are there signs of sample ratio mismatch?
   - Could novelty effect or seasonality explain the result?

4. SEGMENTATION
   - Suggest 3 segments worth analyzing (device, source, new vs returning)
   - Could the result be driven by one segment?

5. DECISION & NEXT STEPS
   - Ship it / Kill it / Keep testing — with clear reasoning
   - If keep testing: what to change and required sample size
   - What follow-up test would you recommend?

Be rigorous. Do not let me make a decision on noisy data.
#data#analytics#interpret#test#results

Works with

chatgptclaudegemini

💡 Pro Tips

  • •Always check for sample ratio mismatch — if A and B have very different sample sizes, something went wrong
  • •Statistical significance ≠ practical significance — a 0.01% lift can be "significant" with enough data
  • •Run tests for full weeks (7, 14, 21 days) to avoid day-of-week bias

✨ Example Output

Test: New CTA button ("Start Free" vs "Sign Up")

STATISTICAL SIGNIFICANCE:
- Control: 12,450 visitors → 387 conversions (3.11%)
- Variant: 12,380 visitors → 425 conversions (3.43%)
- Absolute lift: +0.32 percentage points
- Relative lift: +10.3%
- p-value: 0.038 → Statistically significant at 95% (barely)
- 95% CI for difference: [+0.02%, +0.62%]

PRACTICAL SIGNIFICANCE:
- The lower bound of the CI is nearly zero — the true effect could be tiny
- At 100K monthly visitors: ~320 extra conversions/month
- If each conversion = 0 → ~6K/month uplift

VALIDITY CHECK:
⚠️ Test ran 8 days — should run at least 2 full weeks to capture weekly cycles
✅ Sample ratio: 50.1% / 49.9% — no mismatch detected
⚠️ Consider novelty effect for UI changes

DECISION: KEEP TESTING for 1 more week. The result is promising but the confidence interval is wide. If it holds after a full 2-week cycle, ship it.

🧠 Why This Works

This prompt enforces rigorous statistical thinking by requiring confidence intervals, effect sizes, and power analysis—not just p-values. It distinguishes between statistical significance and practical significance, preventing the common mistake of shipping changes with trivially small real-world impact.

📅 When to Use This Prompt

Use after an A/B test completes and you need to decide whether to ship the variant, extend the test, or abandon it. Essential for product managers interpreting experiment results, growth teams evaluating landing page tests, or anyone who needs to explain test outcomes to stakeholders.

🎯 What You'll Get

You get a clear verdict with confidence level, practical impact quantification (revenue or conversion lift in real terms), identification of potential confounders or Simpson's paradox risks, and specific next-step recommendations including whether to iterate or move on.

🔗 Related Prompts

Marketing & GrowthPremium

Marketing Funnel Optimizer

Diagnose leaks in your marketing funnel and get specific fixes for every stage — from awareness to purchase.

funnelconversioncro
★4.8
advanced