Product Experiment Decision Gate

The problem

Product experiments can produce clean-looking results without producing trustworthy decisions. A metric goes up, the team calls the test a win, and important questions about proxy quality, guardrails, contamination, measurement, and false positives are left unresolved.

What it does

• Checks whether an experiment is actually capable of supporting the product decision attached to it.
• Separates the hypothesis, proposed mechanism, and current evidence so assumptions do not silently become proof.
• Audits primary metrics for proxy risk and identifies meaningful guardrails against hidden product or operational harm.
• Finds instrumentation gaps, contamination risks, population mismatches, novelty effects, and post-hoc interpretation.
• Checks whether sample-size logic, statistical method, thresholds, and stopping rules are actually established rather than assumed.
• Tests the cost of false positives and false negatives before recommending SHIP, ITERATE, STOP, or RE-TEST.

Why this beats prompting it yourself

Generic experiment prompts are usually good at generating test ideas, but weaker at challenging whether the test can actually justify the decision that follows. This skill pressure-tests the decision logic underneath the experiment and refuses to invent baselines, sample sizes, thresholds, or statistical certainty just to make the plan look complete.

Use cases

• Review A/B tests before launch to find weak hypotheses, bad success metrics, or missing guardrails.
• Audit onboarding, pricing, growth, feature, or UX experiments before rollout decisions.
• Check whether a positive result is strong enough to support shipping or scaling.
• Find false-pass pathways where a test can look successful while creating downstream harm.
• Review completed experiment results for post-hoc interpretation, confounds, or decision-rule drift.

Known limitations

This skill does not calculate statistical power without the required inputs, certify causal validity, independently validate analytics instrumentation, or replace a statistician or experiment analyst when formal statistical review is required. Strong conclusions depend on the quality of the experiment design, data, and context provided.