校验A/B实验结果 · A/B Test Validator

它解决什么问题

团队看到实验「显著」就急着上线,常忽略样本量不足、提前偷看结果、分组不随机等问题;也有人在数据里挑子群体,把噪声当成发现。直接问 AI,常得到「显著就有效」的草率结论。

这个技能做什么

把实验数据与设计描述输入后,按「设计核查 → 数据质量 → 统计复算 → 结论判定」四步校验:检查随机化、样本量、显著性水平、置信区间与多指标偷看风险,输出可信度评级与需要人工确认的清单。

为什么比裸提示词强

• 防误判:强制检查显著性、样本量与偷看风险。
• 可复算:给出核心指标的计算口径与公式。
• 结论分级:区分「可上线」「需继续实验」「数据无效」。

典型用例

• 上线前复核 A/B 实验结论是否成立。
• 审计历史实验数据的统计口径。
• 为增长团队建立实验验收标准。

局限

需要你提供原始数据或汇总指标;无法自动抓取实验平台数据,复杂分层模型建议由数据同事复核。

English · 英文介绍

The problem

When the team sees that the experiment is "significant", they rush to launch it. They often ignore problems such as insufficient sample size, peeking at the results in advance, and non-random grouping. Some people also select subgroups in the data and treat noise as discovery. When asking AI directly, we often get the hasty conclusion that "it is effective if it is obvious".

What it does

After entering the experimental data and design description, press "Design Verification → Data Quality → Statistical Recalculation → Conclusion Judgment" to perform four-step verification: check randomization, sample size, significance level, confidence interval and multi-index peek risk, and output a credibility rating and a list that requires manual confirmation.

Why is it better than naked prompt words?

• Prevent misjudgment: Mandatory checking of significance, sample size and peek risk.
• Recalculable: The calculation caliber and formula of the core indicators are given.
• Conclusion classification: distinguish between "can go online", "need to continue experimentation" and "invalid data".

Typical use cases

• Review whether the A/B experiment conclusions are valid before going online.
• Audit the statistical caliber of historical experimental data.
• Establish experiment acceptance criteria for growth teams.

Limitations

You are required to provide original data or summary indicators; experimental platform data cannot be automatically captured, and complex hierarchical models are recommended to be reviewed by data colleagues.