Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    A/B Test Significance Calculator

    1

    Rigorous A/B test calculator for p-values, z-scores, sample sizes, and 95% confidence intervals.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    A/B Test Significance Calculator

    A/B Test Significance Calculator

    Example session with this skill installed

    Analyze our checkout test. Control: 10,000 visitors, 500 conversions. Treatment: 10,050 visitors, 580 conversions. Is this significant at 95% confidence?

    • Read your context and instructions
    • Compiled the a b test

    I have performed a two-tailed Z-test on the provided checkout experiment data to determine statistical significance.

    A/B Test Statistical Analysis

    MetricControl GroupTreatment Group
    Visitors10,00010,050
    Conversions500580
    Conversion Rate5.00%5.77%
    Standard Error0.002180.00233
    95% Conf. Interval[4.57%, 5.43%][5.31%, 6.23%]

    Statistical Summary

    • Relative Lift: +15.42%
    • Z-Score: 2.418
    • P-Value: 0.0156
    • Result: Statistically Significant Winner

    The p-value (0.0156) is below the alpha threshold of 0.05. The treatment group shows a statistically significant improvement over the control. While there is a slight overlap in the 95% confidence intervals (5.31% to 5.43%), the Z-score confirms the probability that this lift occurred by chance is less than 2%.

    Next steps

    1. Document the 15.4% lift and proceed with a full rollout to 100% of traffic.
    2. Monitor the backend checkout performance to ensure the conversion lift translates to completed revenue events.
    3. Archive the results in the experiment log to inform future baseline conversion expectations.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Inconclusive A/B tests and miscalculated sample sizes waste engineering resources and lead to false positives. This skill provides rigorous statistical validation for experimentation data, ensuring every product decision is backed by hard math rather than gut feeling.

    What it does

    • Sample Size Estimation determines the required number of users per variant based on baseline rates and desired power.
    • Statistical Significance calculates two-tailed Z-scores and P-values to validate experimental results.
    • Confidence Intervals establishes 95% CI boundaries for both control and treatment variants to visualize overlap.
    • MDE Calculation identifies the minimum detectable effect needed to reach significance within a given timeframe.
    • Outcome Classification labels tests as Statistically Significant, Inconclusive/Underpowered, or Negative.

    How it works

    1. Provide the skill with your baseline conversion rate, alpha (default 0.05), and target power (default 0.80).
    2. Input the observed trial counts and conversion counts for both control and treatment groups.
    3. The skill runs a deterministic statistical workflow to calculate Z-scores and P-values.
    4. Review the generated report to determine if the test reached the significance threshold.

    Frameworks & tools

    This skill operates within workspace-confined environments and is compatible with Claude Code, Cursor, Windsurf, and Gemini CLI. It relies on standard statistical models for two-tailed Z-tests.

    Why this beats prompting it yourself

    Generic prompts often fail to account for the "peeking problem" or miscalculate standard error for conversion rates. This skill follows a fixed experimentation workflow that enforces 95% confidence intervals and 80% statistical power standards.

    Use cases

    • Pre-test planning to determine how long a feature flag should run.
    • Post-test analysis to verify if a conversion lift is statistically valid.
    • Audit existing test results to identify underpowered experiments.

    Known limitations

    Calculations are restricted to frequentist statistics for two-tailed Z-tests. It does not support Bayesian analysis or complex multi-variant (A/B/n) adjustments like Bonferroni correction.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.

    Supports frequentist Z-tests only. Does not support Bayesian inference or multi-variant (A/B/n) correction methods. Restricted to workspace file access.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed today

    Linux only

    Software
    Zero global installation
    orelevated privileges required.

    File System Read/Write (strictly scoped to current project workspace `./`).

    Permissions

    Read Files

    Target Use Cases: - Calculate statistical significance for an A/B test. - Determine sample size needed for 95% confidence A/B test. - Evaluate if my conversion rate lift is statistically significant.

    Listedtoday

    What's inside

    Frequently Asked Questions

    This skill calculates sample sizes, p-values, z-scores, and 95% confidence intervals to ensure your experiments are statistically valid and not the result of random chance.

    The skill is designed for developer-focused AI environments including Claude Code, Cursor, Windsurf, and Gemini CLI, functioning seamlessly within your existing coding workspace.

    The skill currently focuses on frequentist statistics using two-tailed Z-tests for conversion rates; it does not support Bayesian models or multi-variant adjustments like the Bonferroni correction.

    While generic prompts can hallucinate math or ignore standard error, this skill follows a deterministic statistical workflow to ensure 95% confidence intervals are calculated accurately every time.

    You get the full statistical engine for pre-test planning (sample size/MDE) and post-test analysis (significance/confidence intervals), along with any future updates or refinements to the calculation logic.

    New in Testing & Debugging