More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Experiment Design: A/B Tests That Answer

    1

    Design A/B tests that end in a decision: a falsifiable hypothesis, a pre-committed contract, honest sample sizing, and a ship, kill, or inconclusive verdict you follow.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Experiment Design: A/B Tests That Answer

    Experiment Design: A/B Tests That Answer

    Example session with this skill installed

    We want to test simplifying our pricing page — one recommended plan instead of the three-tier grid. Conversion is about 12% and we get roughly 4,000 visitors a day. Design the test.

    • Read your context and instructions
    • Compiled the experiment design

    PRE-COMMITMENT CONTRACT — produced by applying the skill.

    Test: pricing page simplification. Written and timestamped before launch; stored where it cannot be edited silently.

    Hypothesis: replacing the three-tier comparison grid with one recommended tier (alternatives behind a "compare plans" toggle) will lift signup-to-paid conversion 8% relative — 12.0% to about 13.0% — by cutting decision friction for visitors who already intend to subscribe. Falsifier recorded: we will not ship if the refund guardrail trips, even on a clear conversion win.

    Primary metric: signup-to-paid conversion. Guardrails, each with a trip threshold: refund rate within 14 days (+0.3 points trips), support tickets per signup (+10% relative trips), page latency p75 (+200 ms trips), revenue per signup (−3% relative trips).

    Sizing: MDE 1.0 percentage point at 80% power, alpha 0.05 two-sided, needs about 17,500 visitors per arm. At roughly 2,000 per arm per day that is about 9 days of sample — but UI changes carry a 14-day floor for novelty and primacy effects, so duration is set to 14 days, covering two full weekly cycles. Feasibility check passed: required sample arrives well inside the six-week ceiling.

    Decision rule, fixed now: ship if the confidence interval excludes zero upward and every guardrail holds; kill if it excludes zero downward or any guardrail trips; otherwise the result is inconclusive and the default is kill — a revisit requires a bolder redesign under a new contract. No segment analysis beyond what is declared here. Analysis happens once, on day 15. Daily peeking is not analysis, and "directional" is not a verdict.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most experiment programs produce arguments, not answers: vague hypotheses nobody can falsify, daily peeking until something turns green, guardrails checked after the ship decision, and "directional but not significant" treated as evidence. This skill replaces all of that with one artifact — the pre-commitment contract, written before launch, timestamped somewhere it cannot be quietly edited, and followed mechanically at readout. It starts with hypothesis craft: cause, effect, magnitude, and mechanism, plus a falsifiability test that kills experiments whose outcome cannot change the decision. Then sizing as arithmetic, not hope: the inverse-square relationship between detectable effect and sample, an 80% power floor, and a feasibility decision point that stops tests whose required sample cannot arrive inside the duration ceiling — with the honest alternatives named (bolder change, wider eligibility, or a different method entirely). Duration rules cover weekly cycles, the 14-day floor for UI changes, novelty and primacy effects, and long-term holdouts for permanent changes. Running discipline covers peeking and false-positive inflation, binding guardrails with numeric trip thresholds, concurrent-test interference, ratio-metric variance, and sample-ratio mismatch as a stop signal. Readout resolves into three buckets — ship, kill, inconclusive — with ranked responses for the inconclusive case and a hard rule that post-hoc segments are noise until a confirmatory test says otherwise. You get the fill-in contract template, a fully worked pricing-page example with real sizing numbers, a design readiness rubric, a pre-launch checklist, and edge-case handling for network effects, low traffic, and contaminated runs.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Research & Analysis.

    • It cannot manufacture traffic: if your volume cannot support the MDE inside the duration ceiling, it will tell you the test is not worth running rather than design a weaker one quietly.
    • It assumes standard frequentist analysis; teams committed to Bayesian or sequential frameworks keep the contract discipline but should swap in their own readout math.
    • Network-effect products (marketplaces, social graphs) break plain user-level randomization — the skill flags this and points to cluster or geographic designs, but those need platform support it cannot provide.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 24 days ago

    Permissions

    No special permissions declared or detected

    Listed24 days ago
    Updated21 days ago

    What's inside

    Frequently Asked Questions

    Usually one of three things: the change is too small for your traffic (the sizing step exposes this before launch — sample grows with the inverse square of the effect), the test was stopped or read early, or the hypothesis never named a magnitude, so nobody can tell a miss from an underpowered run. The contract forces all three into the open before a single user is randomized.

    Not with fixed-horizon statistics — daily peeking with a stop-at-first-significance rule pushes the false-positive rate toward 30%, so about a third of your "wins" would be noise. Either pre-commit to one analysis date and honor it, or use sequential methods designed for continuous monitoring. The skill's running-discipline section covers both routes.

    Not on that evidence. Segments declared in the contract before launch are evidence; segments discovered afterward are noise mining — test twenty segments at the usual threshold and one wins by chance alone. The skill's rule: a discovered segment earns a confirmatory test with its own contract, and ships only if that test confirms it.

    New in Research & Analysis