- Home
- Skills
- Trading & Investing
- financial ml trading backtest
Works with the AI Agents you already use
financial ml trading backtest
Methodology for training and validating financial ML models with leakage prevention and overfitting detection.
$19
financial ml trading backtest
Example session with this skill installed
Review my training pipeline for a BTC/USDT 1h strategy. I'm using random 5-fold CV and getting 72% accuracy, but the backtest in my engine shows a negative Sharpe. Why the discrepancy?
- Read your context and instructions
- Compiled the financial ml trading
The discrepancy is due to two critical failures: random k-fold CV on time series causes massive data leakage, and accuracy is a deceptive metric for trading as it ignores costs and class balance. Your 72% accuracy likely reflects the model learning the majority 'Hold' class or 'peeking' at future price action through the random folds.
I have analyzed your pipeline and mapped the required changes to meet the Financial ML standard.
Pipeline Audit and Remediation
| Component | Current State | Required Correction | Impact |
|---|---|---|---|
| Validation | Random 5-Fold | Walk-forward with Embargo | Eliminates look-ahead bias and temporal leakage. |
| Metric | Accuracy (72%) | Net Sharpe / Profit Factor | Accounts for 0.1% fees and slippage per trade. |
| Labeling | Simple Direction | Triple-Barrier Method | Includes SL/TP and time-decay logic. |
| Leakage | Not Checked | Shift Test + Purging | Ensures features at time T only use data <= T. |
| Overfitting | Ignored | DSR & PBO Calculation | Corrects Sharpe for the number of trials attempted. |
Proposed Validation Strategy
# Transitioning from random K-Fold to Purged Walk-Forward
from finml.validation import WalkForwardEmbargo
# 1h candles require embargo equal to the label horizon
splitter = WalkForwardEmbargo(
n_folds=5,
embargo_bars=24, # 24h embargo for 1h labels
purge_size=0.05
)
# Training must use Financial Objective
def objective(trial):
# Train fold...
# Generate OOS predictions...
# Return net_sharpe(returns, fees=0.001)
Next steps
- Run the Shift Test on your features to confirm no future data is leaking into the training rows.
- Implement Triple-Barrier labeling using a threshold of at least 0.2% (2x your round-trip cost).
- Re-run validation using the Purged Walk-Forward splitter and report the Net Sharpe.
Connects securely to your tools. The creator never sees your data.
About this skill
Building trading models with random k-fold and accuracy metrics leads to catastrophic failure in live markets. Standard ML assumptions break when applied to non-stationary, autocorrelated financial time series where execution costs erode all theoretical gains. This skill enforces a rigorous methodology for training, validating, and backtesting machine learning strategies that actually account for market reality.
What it does
- Data Quality Gates Enforces OHLCV validation including gap detection, volume spike filtering, and stationary feature transforms.
- Leakage Prevention Implements walk-forward validation with purging and embargo periods to prevent temporal data leakage.
- Financial Labeling Uses triple-barrier methods and meta-labeling with cost-aware thresholds rather than simple direction labels.
- Overfitting Detection Calculates Deflated Sharpe Ratio (DSR) and Probability of Backtest Overfitting (PBO) to adjust for multiple comparisons.
- Realistic Backtesting Replays production logic bar-by-bar with taker fees, slippage tiers, and order-book simulation.
- Drift Monitoring Tracks Population Stability Index (PSI) and ECE calibration to detect model decay before losses mount.
How it works
- Validation Setup Configure the walk-forward splitter with an embargo period equal to the label horizon to ensure out-of-sample integrity.
- Feature Engineering Apply ADF tests to ensure all inputs are stationary and run shift tests to verify no look-ahead bias exists.
- Training & Search Run hyperparameter optimization using financial objectives like Net Sharpe instead of loss functions.
- Evaluation Generate the standardized report comparing IS vs OOS performance and calculating the three mandatory overfitting metrics.
Frameworks & tools
Python-based workflows utilizing libraries for time-series analysis and financial modeling. Works with OHLCV data from providers like Binance, Bybit, and Alpaca.
Why this beats prompting it yourself
Generic AI prompts often suggest k-fold cross-validation or SMOTE, which are fatal for time-series trading. This skill embeds specific financial ML defenses, like embargoed purging and DSR, that prevent the model from "solving" the backtest through noise or leakage.
Use cases
- Validating if a crypto bot's high backtest Sharpe is actually due to overfitting or data leakage.
- Implementing a robust walk-forward training pipeline for equities that accounts for delistings and gaps.
- Building a drift-monitoring dashboard to alert when a live strategy's features deviate from training distributions.
- Comparing multiple model architectures using PBO to select the one least likely to fail out-of-sample.
Known limitations
Does not handle exchange API integrations, order routing, or tax reporting. Performance is constrained by the quality of historical OHLCV data provided.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Trading & Investing.
No exchange API/execution logic. No portfolio accounting or tax reporting. Requires minimum 90-day OHLCV history for valid gates. Stationary transforms are mandatory for all inputs.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 11 days ago