Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    ab test design and evaluation

    1

    This skill automates the statistical rigor required for two-arm randomized controlled trials, ensuring your A/B tests are mathematically sound from design to evaluation.

    $2.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    ab test design and evaluation

    ab test design and evaluation

    Example session with this skill installed

    Design a two-arm test for a new landing page. Baseline conversion is 12%. I want to detect a 5% relative MDE with 80% power at 0.05 alpha. We get 5000 eligible units per day. Dataset ref: E1-LP-TEST.

    • Read your context and instructions
    • Compiled the ab test design

    Analysis of the requested experiment design is complete. The sizing requirements and run duration have been calculated.

    A/B Test Report — Landing Page Conversion Optimization, 2023-11-24

    1. Scope, sources and assumptions

    • Sources: E1 (User supplied design parameters).
    • Dataset Reference: E1-LP-TEST (Approved by: user).
    • Randomization Unit: user_id.
    • Primary Metric: landing_page_conversion (type: proportion, direction: increase).
    • Assumptions: Baseline value of 0.12 is treated as the true control value.

    2. Experiment plan

    ParameterValue
    Units per Arm134,812
    Total Units Required269,624
    Alpha (per look)0.0500
    Power0.8000
    Absolute MDE0.0060 (5.0% relative)
    Estimated Run Length54 days

    3. Randomization and validity audit

    No results to report. Randomization audit requires inputs.results which were not supplied for this sizing request.

    4. Treatment effect and guardrails

    No results to report. Effect evaluation was not requested.

    5. Peeking, predeclaration and data-access checks

    • Looks: 0 observed / 1 planned.
    • Predeclared Endpoints: landing_page_conversion (confirmed).
    • Access Scope: E1-LP-TEST (Authorized).

    6. Warnings and information gaps

    • Information gap — organizational validation required: Deployment feasibility for 54-day duration.

    7. Decision support and items for human review

    • Analytical estimate — validation required: The required duration of 54 days may exceed typical sprint cycles.
    • Management decision required: Approve experiment start based on 270k unit requirement.
    • Proposed Action: Proceed with implementation if 54-day window is acceptable.

    This analysis supports, and does not replace, professional judgment and management's decision.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Stop guessing at experiment significance or manual sample size calculations. This skill automates the statistical rigor required for two-arm randomized controlled trials, ensuring your A/B tests are mathematically sound from design to evaluation. It handles the heavy lifting of power analysis and post-test auditing through a deterministic execution environment.

    What it does

    • Size experiments by computing required sample sizes and power for proportion or mean metrics based on your MDE.
    • Audit randomization via sample ratio mismatch (SRM) checks, chi-square tests, and covariate balance verification.
    • Detect validity issues such as data leakage, contamination, peeking, and missing consent exclusions.
    • Evaluate effects by calculating treatment lift, confidence intervals, and p-values for primary and guardrail metrics.
    • Check predeclarations to ensure analysis aligns with original experiment plans and look-schedules.

    How it works

    1. Prepare a request.json containing your experiment objective, metrics, baseline data, and dataset approval status.
    2. The agent runs a deterministic Python script to perform all arithmetic and statistical tests.
    3. Review the result envelope which flags warnings, SRM failures, or power deficiencies.
    4. Receive a structured Markdown report containing design tables, validity audits, and causal interpretations.

    Frameworks & tools

    The skill relies on a self-contained Python execution environment using a specialized script for exact statistical computation. It interfaces with JSON schemas for input validation and generates GitHub-flavored Markdown reports.

    Why this beats prompting it yourself

    Generic LLM prompts often hallucinate p-values or fail at complex power calculations. This skill uses a deterministic script for arithmetic, ensuring every number is traceable, unrounded, and mathematically accurate according to experiment design standards.

    Use cases

    • Determine how many days a new checkout flow needs to run to detect a 2% lift in conversion.
    • Audit a completed marketing test to see if a lopsided sample ratio invalidated the results.
    • Compare treatment vs. control metrics while monitoring guardrail targets for regressions.
    • Verify if "peeking" at data early has compromised the effective alpha of your experiment.

    Known limitations

    Does not support multi-arm, factorial, bandit, or Bayesian analyses. Limited to two-arm experiments using proportion or mean metrics. Requires pre-approved dataset access and explicit MDE inputs.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse All Claude Skills.

    Supports only two-arm tests. No Bayesian, multi-arm, or factorial support. Requires deterministic Python script execution. No ratio-metric analysis.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Environment Variables
    Listed3 days ago

    What's inside

    Frequently Asked Questions

    This skill replaces manual spreadsheets and error-prone LLM math with a deterministic Python execution environment that calculates sample size, power, and significance using exact statistical formulas.

    The skill is designed for two-arm randomized controlled trials and supports metrics based on either proportions (e.g., conversion rates) or means (e.g., average order value).

    The skill performs a Chi-square test for Sample Ratio Mismatch (SRM) and verifies covariate balance to ensure your randomization process wasn't compromised by data leakage or technical bugs.

    The purchase includes the structured JSON schema for input, the specialized Python execution script for statistical computation, and a template for the Markdown evaluation reports.

    No, this skill is focused on frequentist statistics for two-arm trials; it currently does not support multi-arm (A/B/n) tests, multi-variate factorial designs, or Bayesian inference.

    Simply provide a JSON file containing your baseline data, Minimum Detectable Effect (MDE), and experiment objectives; the skill then automates the arithmetic and report generation.