More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    A/B Test Stats: Sample size, p-value and SRM check

    1

    Is that conversion lift real, or noise? Plan sample size before the test, then get the p-value, confidence interval and a sample-ratio-mismatch warning from your counts. Plain Python, no stats library.

    $12

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    A/B Test Stats: Sample size, p-value and SRM check

    Example session with this skill installed

    Our synthetic checkout test finished: control 480 conversions out of 5,000, treatment 560 out of 5,000. Check significance and SRM using references/example_input.json.

    • Read your context and instructions
    • Compiled the a b test

    Actual stdout from python3 -I -B scripts/abstats.py analyze --input references/example_input.json (exit 0):

    {
      "command": "analyze",
      "test": "pooled two-proportion z-test",
      "alternative": "two-sided",
      "alpha": 0.05,
      "control": {
        "conversions": 480,
        "n": 5000,
        "rate": 0.096,
        "wilson_ci": [
          0.08814189353,
          0.1044784096
        ]
      },
      "treatment": {
        "conversions": 560,
        "n": 5000,
        "rate": 0.112,
        "wilson_ci": [
          0.1035547974,
          0.1210409393
        ]
      },
      "difference": 0.016,
      "ci_difference": [
        0.004033175968,
        0.02797866526
      ],
      "ci_method": "newcombe",
      "relative_lift": 0.1666666667,
      "ci_relative_lift": [
        0.03947055612,
        0.3094272879
      ],
      "z_statistic": 2.620712092,
      "p_value": 0.00877463306,
      "significant": true,
      "confidence_level": 0.95,
      "srm": {
        "observed": [
          5000,
          5000
        ],
        "expected": [
          5000.0,
          5000.0
        ],
        "allocation": [
          0.5,
          0.5
        ],
        "observed_share": [
          0.5,
          0.5
        ],
        "chi_square": 0.0,
        "df": 1,
        "p_value": 1.0,
        "threshold": 0.001,
        "mismatch": false
      },
      "warnings": []
    }
    

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem Your checkout test finished with 480 conversions out of 5,000 control users and 560 out of 5,000 treatment users. Before you call it a win, you want to know how sure you can be about that gain, and whether the traffic split was clean enough to trust the comparison at all. For the next test, you'd like a sample size before any results come in.

    See it before you buy: real input, real output Command (from the package folder, exit code 0, 0.025 s):

    python3 -I -B scripts/abstats.py analyze --input references/example_input.json
    

    Before: Your counts (excerpt of references/example_input.json, synthetic numbers)

      "control": {"n": 5000, "conversions": 480},
      "treatment": {"n": 5000, "conversions": 560},
      "allocation": [0.5, 0.5],
      "alpha": 0.05,
    

    After: lines copied from the script's output (excerpt):

        "rate": 0.096,
        "rate": 0.112,
      "relative_lift": 0.1666666667,
      "p_value": 0.00877463306,
      "significant": true,
        "mismatch": false
    

    What the numbers show

    • Matches published examples: the statsmodels docs (5/83 vs 12/99, p = .159) and an OpenStax textbook example (20/200 vs 12/200, p = .140).
    • Plan example: 10% baseline, detect a 2-point gain gives 3,841 users per arm.
    • 16 results recomputed separately from the formulas all matched to about 9 significant digits; all 16 runs took about half a second.

    What you get

    • One Python script with three commands: sample-size plans how many users you need, analyze tests observed conversion counts, and srm checks whether your traffic split matches the plan.
    • Planning for conversion rates, or for averages if you know the standard deviation. It handles uneven splits like 2:1 and can turn daily traffic into an estimated number of days.
    • For results: the p-value, a 95% range for each arm's rate, a range for the difference and a range for the relative lift. Everything comes back as JSON.
    • Instructions that make the agent check the split before it talks about lift, and ask you whether "10%" means relative lift or percentage points before it calculates anything.
    • Formula notes, sample inputs, 28 runnable test cases and 2 saved outputs you can compare against.

    Measured on real data (full detail) The two outside examples are published teaching problems, not real customer experiments. The planning and traffic-split examples are made up.

    What I checkedResult
    statsmodels docs example: 5/83 vs 12/99p = .159, same as the docs
    OpenStax textbook example: 20/200 vs 12/200p = .140, same as the book
    Plan: 10% baseline, detect a 2-point gain3,841 users per arm
    16 results recomputed separately from the formulasAll 16 matched to about 9 significant digits
    SpeedAll 16 runs together took about half a second, Python startup included

    references/benchmarks.md lists every command so you can rerun them. The script is a single file under 20 KB and runs without the rest of the package.

    Known limitations

    • It assumes you look at results once, at the planned end. No early stopping rules, exact tests or Bayesian analysis.
    • Averages such as revenue per user can be planned but not analyzed. Ratio metrics and clustered designs need a different method.
    • No correction for testing many metrics at once, and no variance reduction (CUPED). Very small counts need a closer look.
    • The split check tells you something is off, not what caused it. Aggregate counts can't prove randomization worked.

    Who it's for Product analysts and engineers who work with aggregate conversion counts and want repeatable local calculations with an explicit assignment check.

    Requires Python 3.8+ and a host that can run a local Python script. No third-party packages. Counts must be integers, and file input must be UTF-8 JSON.

    Privacy: runs locally, no network, no telemetry, writes nothing unless you redirect output.

    Check it yourself 28 test cases and 2 saved sample outputs pass, and 16 results were recomputed by hand from the formulas. To check it yourself from the package folder, run python3 -I -B scripts/abstats.py analyze --input references/example_input.json and compare the output with references/sample-output/conversion.json. Then run python3 -I -B scripts/abstats.py srm --input references/example_srm_fail.json, which should flag the split and exit 1.

    Price and guarantee $12 one time, yours to use with any agent that can run Python. Covered by Agensi's money-back guarantee if it doesn't do what this page promises.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Marketing & SEO.

    Fixed-horizon only; no sequential monitoring or Bayesian tests. No support for revenue/ratio metrics or SQL extraction. Requires Python 3.8+ standard library.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 15 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed15 days ago
    Updated5 days ago

    What's inside

    Frequently Asked Questions

    They're great for one-off checks in a browser. This gives you the same kind of math as JSON your agent can run locally, plus a split check that exits 1 when assignment looks broken. It isn't more accurate, and small differences in conventions between calculators are normal.

    No. These numbers assume you look once, at the planned end. If you want to check repeatedly and stop early, you need a sequential method, which this doesn't do.

    No. Exit 1 only means the traffic split doesn't match the plan. A result that isn't significant still exits 0 and reports `"significant": false`.

    Agensi's money-back guarantee covers every paid skill: if it doesn't do what this page promises, you get your money back. Everything above is output from the script in the package, so you can see what it does before you buy.

    New in Marketing & SEO