backtest truth detector

The problem

Most backtests look great on paper but fail in live markets due to curve-fitting, look-ahead bias, and ignored execution costs. Developers and traders often waste capital on strategies that rely on unrealistic fills or statistically insignificant samples.

What it does

• Detects look-ahead bias and unrealistic execution assumptions like mid-price fills on large orders.
• Audits sample sizes and market regimes to identify weak statistical significance or survival bias.
• Calculates the impact of missing costs including slippage, bid/ask spreads, and borrow fees.
• Analyzes parameter sensitivity to find evidence of over-optimization and fragile performance spikes.

Why this beats prompting it yourself

General LLMs often take backtest data at face value or miss subtle technical flaws like point-in-time constituent errors. This skill applies a systematic forensic framework that forces a recomputation of metrics under stress, ensuring you see the "net-of-everything" reality before deploying code.

Use cases

• Audit a vendor's high-Sharpe strategy before purchasing or licensing.
• Validate a new quantitative model against out-of-sample data and market impact.
• Identify why a live strategy is diverging from its historical performance.
• Stress-test a portfolio for fragility by removing top outlier trades.

Known limitations

Requires detailed trade logs or strategy parameters to provide a high-confidence verdict. It cannot fix a broken strategy, only identify the flaws in its current state.