overfitting strategy bias auditor

The problem

Data science and trading strategies often fail in production because backtests were inadvertently tuned to noise. Developers struggle to spot subtle data leakage, look-ahead bias, and circular logic that make results appear artificially perfect during evaluation.

What it does

• Identifies data leakage in time-series splits and feature engineering pipelines.
• Detects multiple comparison bias when too many strategy variants are tested on the same dataset.
• Flags weak assumptions regarding stationarity, label quality, and environmental stability.
• Exposes misleading metrics, such as reporting relative lift without absolute base rates.
• Generates a structured audit report with severity ratings and required remediation steps.

Why this beats prompting it yourself

Generic prompts often miss the "unknown unknowns" of statistical bias. This skill applies a systematic adversarial mindset that explicitly hunts for specific red flags like random splits on temporal data or global normalization before splitting. It forces a rigorous reconstruction of the causal mechanism that a standard LLM conversation would likely skip.

Use cases

• Auditing a trading bot backtest before deploying capital.
• Reviewing a machine learning model's evaluation plan for data leakage.
• Validating A/B test results to ensure significance isn't gamed by cherry-picking time windows.
• Sanity checking a new analytics strategy before presenting to stakeholders.

Known limitations

Requires the user to provide detailed methodology and data handling procedures to be effective. It cannot execute code to verify claims; it audits the logic and plan described.