- Home
- Skills
- Testing & Debugging
- A/B Test Significance Calculator
Works with the AI Agents you already use
A/B Test Significance Calculator
Rigorous A/B test calculator for p-values, z-scores, sample sizes, and 95% confidence intervals.
$15
A/B Test Significance Calculator
Example session with this skill installed
Analyze our checkout test. Control: 10,000 visitors, 500 conversions. Treatment: 10,050 visitors, 580 conversions. Is this significant at 95% confidence?
- Read your context and instructions
- Compiled the a b test
I have performed a two-tailed Z-test on the provided checkout experiment data to determine statistical significance.
A/B Test Statistical Analysis
| Metric | Control Group | Treatment Group |
|---|---|---|
| Visitors | 10,000 | 10,050 |
| Conversions | 500 | 580 |
| Conversion Rate | 5.00% | 5.77% |
| Standard Error | 0.00218 | 0.00233 |
| 95% Conf. Interval | [4.57%, 5.43%] | [5.31%, 6.23%] |
Statistical Summary
- Relative Lift: +15.42%
- Z-Score: 2.418
- P-Value: 0.0156
- Result: Statistically Significant Winner
The p-value (0.0156) is below the alpha threshold of 0.05. The treatment group shows a statistically significant improvement over the control. While there is a slight overlap in the 95% confidence intervals (5.31% to 5.43%), the Z-score confirms the probability that this lift occurred by chance is less than 2%.
Next steps
- Document the 15.4% lift and proceed with a full rollout to 100% of traffic.
- Monitor the backend checkout performance to ensure the conversion lift translates to completed revenue events.
- Archive the results in the experiment log to inform future baseline conversion expectations.
Connects securely to your tools. The creator never sees your data.
About this skill
Inconclusive A/B tests and miscalculated sample sizes waste engineering resources and lead to false positives. This skill provides rigorous statistical validation for experimentation data, ensuring every product decision is backed by hard math rather than gut feeling.
What it does
- Sample Size Estimation determines the required number of users per variant based on baseline rates and desired power.
- Statistical Significance calculates two-tailed Z-scores and P-values to validate experimental results.
- Confidence Intervals establishes 95% CI boundaries for both control and treatment variants to visualize overlap.
- MDE Calculation identifies the minimum detectable effect needed to reach significance within a given timeframe.
- Outcome Classification labels tests as Statistically Significant, Inconclusive/Underpowered, or Negative.
How it works
- Provide the skill with your baseline conversion rate, alpha (default 0.05), and target power (default 0.80).
- Input the observed trial counts and conversion counts for both control and treatment groups.
- The skill runs a deterministic statistical workflow to calculate Z-scores and P-values.
- Review the generated report to determine if the test reached the significance threshold.
Frameworks & tools
This skill operates within workspace-confined environments and is compatible with Claude Code, Cursor, Windsurf, and Gemini CLI. It relies on standard statistical models for two-tailed Z-tests.
Why this beats prompting it yourself
Generic prompts often fail to account for the "peeking problem" or miscalculate standard error for conversion rates. This skill follows a fixed experimentation workflow that enforces 95% confidence intervals and 80% statistical power standards.
Use cases
- Pre-test planning to determine how long a feature flag should run.
- Post-test analysis to verify if a conversion lift is statistically valid.
- Audit existing test results to identify underpowered experiments.
Known limitations
Calculations are restricted to frequentist statistics for two-tailed Z-tests. It does not support Bayesian analysis or complex multi-variant (A/B/n) adjustments like Bonferroni correction.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.
Supports frequentist Z-tests only. Does not support Bayesian inference or multi-variant (A/B/n) correction methods. Restricted to workspace file access.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed today
Linux only
- Software
- Zero global installationorelevated privileges required.
File System Read/Write (strictly scoped to current project workspace `./`).
Permissions
Target Use Cases: - Calculate statistical significance for an A/B test. - Determine sample size needed for 95% confidence A/B test. - Evaluate if my conversion rate lift is statistically significant.