Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Trading Strategy Overfitting Detector

    1

    Detect selection bias, parameter overfitting, and backtest fragility in trading strategies using a formal audit framework.

    $9.99

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Trading Strategy Overfitting Detector

    Trading Strategy Overfitting Detector

    Example session with this skill installed

    Audit this EMA crossover strategy. IS Sharpe is 2.1, OOS is 0.4. I tested 500 parameter combinations across 10 years of SPY data. The best setting was a 12/26 EMA. Here is the optimization table.

    • Read your context and instructions
    • Compiled the trading strategy overfitting

    The audit identifies High Overfitting Risk for the submitted EMA strategy. The extreme degradation between In-Sample (IS) and Out-of-Sample (OOS) metrics, combined with a high trial count, suggests the result is a selection artifact.

    ## Trading Overfitting Analysis Report

    CategoryScoreFinding
    Final Verdict34/100High Overfitting Risk
    Research Provenance5/15High trial count (500) without multiple-testing adjustment.
    Parameter Robustness4/15Sharp optimum at 12/26; performance drops 60% at 13/27.
    OOS Integrity8/15Significant IS/OOS divergence suggests OOS was not a true holdout.
    Rank Stability3/10Top 5% of IS candidates failed to maintain top-quartile rank OOS.
    Concentration9/10Profits distributed across multiple regimes; low tail dependence.

    Analysis Highlights

    Parameter Surface: The 12/26 setting sits on a "parameter cliff." Moving the fast EMA by +1 period results in a 42% reduction in Sharpe, indicating the strategy is tuned to specific noise rather than a broad signal.

    Winner's Gap: The selected strategy is a 3-sigma outlier compared to the median candidate in the 500-trial sweep. This is a classic indicator of Selection Bias.

    IS → OOS Degradation: Retention is only 19%. A collapse of this magnitude typically invalidates the underlying research hypothesis.

    Search Breadth: Testing 500 combinations on a single asset (SPY) significantly inflates the probability of finding a "lucky" backtest.

    Next steps

    1. Perform a parameter perturbation test to see if any stable plateaus exist in the 500-trial universe.
    2. Test the 12/26 logic on an uncorrelated asset class (e.g., Gold or Treasury Futures) to check for universal edge.
    3. Conduct a fresh walk-forward analysis with strictly partitioned training and testing windows.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The problem

    Trading strategies often perform well in backtests only to fail in live markets. This happens because researchers unknowingly select historical winners from noise through excessive parameter tuning and dataset reuse.

    What it does

    • Analyzes research provenance to quantify the actual search breadth behind a strategy.
    • Identifies parameter "cliffs" where small changes in settings lead to catastrophic performance collapse.
    • Audits OOS (Out-of-Sample) integrity to detect holdout contamination and dataset reuse.
    • Evaluates rank stability and profit concentration to determine if an edge is structural or accidental.
    • Provides a standardized Trading Overfitting Robustness Score based on evidence-backed criteria.

    Why this beats prompting it yourself

    General LLMs tend to be overly optimistic about backtest results and miss subtle selection biases. This skill uses a specialized diagnostic framework that forces a skeptical reconstruction of the research process, preventing the "hallucination of alpha" that occurs when simple prompts ignore multiple-testing risks.

    Use cases

    • Vetting a new quantitative strategy before committing capital.
    • Auditing a vendor's backtest claims for signs of hidden parameter optimization.
    • Identifying if a strategy's returns are dangerously concentrated in a few lucky trades.
    • Verifying the validity of walk-forward validation results.

    Known limitations

    Does not compute Deflated Sharpe Ratio (DSR) or Probability of Overfitting (PBO) natively, though it assesses their applicability. Requires historical performance data to be provided by the user.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Trading & Investing.

    Does not natively compute DSR/PBO/CSCV. Requires user-provided performance data and research history. Cannot generate live signals or guarantee future profits.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 months ago

    Software
    Python

    Permissions

    Terminal / Shell
    Listed2 months ago

    What's inside

    Frequently Asked Questions

    This skill identifies 'backtest ghosts' by auditing research provenance, locating parameter cliffs, and detecting selection bias that general LLMs often miss. It transforms raw backtest data into a rigorous robustness audit to prevent you from deploying strategies that only work on paper.

    You will need to provide the strategy's historical performance logs, parameter ranges used during optimization, and the out-of-sample testing results. The tool uses these inputs to reconstruct the search breadth and identify if the results are structural or accidental.

    While it can evaluate the robustness of any quantitative strategy, it is particularly effective for systematic equity, forex, and crypto strategies where multiple-testing bias is a common risk. It is less suited for qualitative, long-term fundamental analysis that lacks a repeatable backtest framework.

    This skill is a configuration for leading AI agents; once purchased, you can deploy it to your preferred agent environment via the provided integration steps. It includes the full diagnostic framework, standardized scoring templates, and a guide on interpreting the Robustness Score.

    No, this skill functions as a skeptical auditor rather than a statistical calculator. It focuses on the qualitative and structural integrity of the research process, though it can interpret DSR and PBO values if you have already calculated them elsewhere.

    New in Trading & Investing