Trading Strategy Backtest Auditor Skill

Backtesting reports often suffer from hidden biases that lead to catastrophic real-world losses. This skill audits your quantitative strategy results to expose look-ahead bias, overfitting, and unrealistic execution assumptions before you commit capital.

What it does

• Integrity checks scans for look-ahead bias, target leakage, survivorship bias, and train/test contamination.
• Expectancy analysis calculates trade counts, win/loss rates, mean net expectancy, and profit factors using verified formulas.
• Risk assessment evaluates peak-to-trough drawdowns, recovery times, and tail loss concentration from equity series data.
• Execution realism stress-tests results against slippage, bid-ask spreads, and financing costs to find the break-even threshold.
• Overfitting detection identifies parameter cliffs, regime concentration, and multiple-testing risks in the optimization process.

How it works

1. Intake Provide your strategy rules, trade logs (CSV), instrument universe, and commission assumptions.
2. Audit The agent reviews the data for methodology integrity, cost realism, and statistical fragility.
3. Verdict Receive an executive verdict on credibility (Credible, Conditionally Credible, or Not Yet Credible).
4. Action Follow the prioritized robustness matrix to fix high-severity leaks and run suggested walk-forward tests.

Frameworks & tools

This skill works with strategy outputs from platforms like QuantConnect, MetaTrader, TradingView, or custom Python/Pandas backtesting engines. It applies quantitative finance principles including walk-forward validation, bootstrap confidence intervals, and purged/embargoed splits.

Why this beats prompting it yourself

Generic prompts often miss subtle data leakage like future-revised data or timezone misalignments. This skill uses a systematic severity-rated framework that forces a skeptical review of every metric, ensuring you don't overlook a critical parameter cliff.

Use cases

• Auditing a vendor's strategy performance claims before purchasing a signal or bot.
• Validating a custom Python backtest to ensure no target leakage occurred during feature engineering.
• Stress-testing a profitable equity curve against 2x slippage and increased borrow costs.
• Preparing a strategy for production by identifying the most robust parameter sets.

Known limitations

Cannot independently verify the raw data source quality or corporate action accuracy if not provided. Does not provide personalized financial advice or guarantee future returns.