Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    financial ml trading backtest

    1

    Methodology for training and validating financial ML models with leakage prevention and overfitting detection.

    $19

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    financial ml trading backtest

    financial ml trading backtest

    Example session with this skill installed

    Review my training pipeline for a BTC/USDT 1h strategy. I'm using random 5-fold CV and getting 72% accuracy, but the backtest in my engine shows a negative Sharpe. Why the discrepancy?

    • Read your context and instructions
    • Compiled the financial ml trading

    The discrepancy is due to two critical failures: random k-fold CV on time series causes massive data leakage, and accuracy is a deceptive metric for trading as it ignores costs and class balance. Your 72% accuracy likely reflects the model learning the majority 'Hold' class or 'peeking' at future price action through the random folds.

    I have analyzed your pipeline and mapped the required changes to meet the Financial ML standard.

    Pipeline Audit and Remediation

    ComponentCurrent StateRequired CorrectionImpact
    ValidationRandom 5-FoldWalk-forward with EmbargoEliminates look-ahead bias and temporal leakage.
    MetricAccuracy (72%)Net Sharpe / Profit FactorAccounts for 0.1% fees and slippage per trade.
    LabelingSimple DirectionTriple-Barrier MethodIncludes SL/TP and time-decay logic.
    LeakageNot CheckedShift Test + PurgingEnsures features at time T only use data <= T.
    OverfittingIgnoredDSR & PBO CalculationCorrects Sharpe for the number of trials attempted.

    Proposed Validation Strategy

    
    # Transitioning from random K-Fold to Purged Walk-Forward
    from finml.validation import WalkForwardEmbargo
    
    # 1h candles require embargo equal to the label horizon
    splitter = WalkForwardEmbargo(
        n_folds=5,
        embargo_bars=24, # 24h embargo for 1h labels
        purge_size=0.05
    )
    
    # Training must use Financial Objective
    def objective(trial):
        # Train fold...
        # Generate OOS predictions...
        # Return net_sharpe(returns, fees=0.001)
    

    Next steps

    1. Run the Shift Test on your features to confirm no future data is leaking into the training rows.
    2. Implement Triple-Barrier labeling using a threshold of at least 0.2% (2x your round-trip cost).
    3. Re-run validation using the Purged Walk-Forward splitter and report the Net Sharpe.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Building trading models with random k-fold and accuracy metrics leads to catastrophic failure in live markets. Standard ML assumptions break when applied to non-stationary, autocorrelated financial time series where execution costs erode all theoretical gains. This skill enforces a rigorous methodology for training, validating, and backtesting machine learning strategies that actually account for market reality.

    What it does

    • Data Quality Gates Enforces OHLCV validation including gap detection, volume spike filtering, and stationary feature transforms.
    • Leakage Prevention Implements walk-forward validation with purging and embargo periods to prevent temporal data leakage.
    • Financial Labeling Uses triple-barrier methods and meta-labeling with cost-aware thresholds rather than simple direction labels.
    • Overfitting Detection Calculates Deflated Sharpe Ratio (DSR) and Probability of Backtest Overfitting (PBO) to adjust for multiple comparisons.
    • Realistic Backtesting Replays production logic bar-by-bar with taker fees, slippage tiers, and order-book simulation.
    • Drift Monitoring Tracks Population Stability Index (PSI) and ECE calibration to detect model decay before losses mount.

    How it works

    1. Validation Setup Configure the walk-forward splitter with an embargo period equal to the label horizon to ensure out-of-sample integrity.
    2. Feature Engineering Apply ADF tests to ensure all inputs are stationary and run shift tests to verify no look-ahead bias exists.
    3. Training & Search Run hyperparameter optimization using financial objectives like Net Sharpe instead of loss functions.
    4. Evaluation Generate the standardized report comparing IS vs OOS performance and calculating the three mandatory overfitting metrics.

    Frameworks & tools

    Python-based workflows utilizing libraries for time-series analysis and financial modeling. Works with OHLCV data from providers like Binance, Bybit, and Alpaca.

    Why this beats prompting it yourself

    Generic AI prompts often suggest k-fold cross-validation or SMOTE, which are fatal for time-series trading. This skill embeds specific financial ML defenses, like embargoed purging and DSR, that prevent the model from "solving" the backtest through noise or leakage.

    Use cases

    • Validating if a crypto bot's high backtest Sharpe is actually due to overfitting or data leakage.
    • Implementing a robust walk-forward training pipeline for equities that accounts for delistings and gaps.
    • Building a drift-monitoring dashboard to alert when a live strategy's features deviate from training distributions.
    • Comparing multiple model architectures using PBO to select the one least likely to fail out-of-sample.

    Known limitations

    Does not handle exchange API integrations, order routing, or tax reporting. Performance is constrained by the quality of historical OHLCV data provided.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Trading & Investing.

    No exchange API/execution logic. No portfolio accounting or tax reporting. Requires minimum 90-day OHLCV history for valid gates. Stationary transforms are mandatory for all inputs.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 11 days ago

    Permissions

    Read Files
    Listed11 days ago

    What's inside

    Frequently Asked Questions

    This skill addresses the 'backtest trap' where standard ML models appear profitable but fail in live markets due to temporal data leakage, non-stationary features, and unrealistic cost assumptions. It implements specialized defenses like walk-forward validation with purging/embargo periods and Deflated Sharpe Ratios to ensure your model's performance is statistically sound.

    The skill works with any Python-based AI agent framework and supports standard OHLCV (Open, High, Low, Close, Volume) data from major exchanges like Binance, Bybit, and Alpaca. It is designed to be data-agnostic, provided your input meets the required financial time-series format.

    You receive the complete methodology for training and validation, including the triple-barrier labeling logic, walk-forward splitting configurations, and the overfitting detection suite (DSR and PBO). It also includes a standardized reporting module to compare in-sample versus out-of-sample performance.

    This skill focuses exclusively on the validation and training methodology for machine learning models; it does not handle live order execution, API connectivity to exchanges, or portfolio tax reporting. It is intended to be used at the research and strategy development stage before a bot is deployed to live markets.

    Yes, the skill is maintained to keep pace with the latest advancements in quantitative finance. Buyers receive access to updates that include new overfitting metrics, improved feature stationary tests, and optimizations for newer Python time-series libraries.

    New in Trading & Investing