Trading Strategy Backtest & Overfitting Auditor

Trading Strategy Backtest & Overfitting Auditor is a premium quantitative strategy-validation skill designed to determine whether a trading backtest deserves to be trusted. It can analyze: Pine Script Strategies Pine Script Indicators Used as Strategy Logic TradingView Strategy Tester Results Natural-Language Trading Strategies Trade Lists CSV Exports Optimization Reports Parameter Sweeps Equity Curves Walk-Forward Tests Multi-Period Tests Multi-Market Tests Forward-Test Results Paper-Trading Results Screenshots Performance Summaries Broker-Simulation Reports The skill does not begin by asking whether the strategy is profitable. It begins by asking whether the test itself is valid. Its operating sequence is: Strategy Logic → Test Integrity → Data Integrity → Execution Realism → Sample Adequacy → In-Sample / Out-of-Sample Review → Selection-Bias Analysis → Parameter Sensitivity → Performance Distribution → Drawdown and Loss Behavior → Regime Robustness → Period Robustness → Stress Testing → Final Classification The skill is designed to detect misleading backtest results caused by: Look-Ahead Bias Repainting Higher-Timeframe Future Leakage Invalid Intrabar Assumptions Unrealistic Same-Bar Fills Synthetic Chart Prices Missing Commission Unrealistic Slippage Ignored Spread Unlimited Liquidity Assumptions Incorrect Position Sizing Unrealistic Leverage Pyramiding Exposure Incorrect Session Logic Insufficient Warmup Data Problems Optimization Bias Selection Bias Multiple Testing Repeated OOS Reuse Low Trade Count Correlated Trades Narrow Parameter Optima Profit Concentration Regime Dependence Period Dependence Cost Fragility Execution Fragility The skill performs Pine Script integrity review. It can inspect: strategy() configuration Initial Capital Quantity Method Default Quantity Pyramiding Commission Slippage Margin Assumptions Signal Timing Entry Logic Exit Logic Stop Logic Target Logic Trailing Stops Partial Exits Reversals Bar Confirmation Higher-Timeframe Data Lower-Timeframe Data Historical vs Realtime Behavior Session Filters Timezone Date Filters Warmup State Initialization Order Processing Same-Bar Execution Alert Logic vs Strategy Logic Repainting is classified as: No Material Repaint Concern Detected Potential Repaint Risk Confirmed Repaint / Future Leakage Unknown The skill gives special attention to higher-timeframe requests because incorrectly aligned higher-timeframe values can create historical signals that use information unavailable at the actual decision point. Confirmed future leakage is treated as a critical test-integrity failure. The skill audits synthetic chart usage. Potential chart types include: Heikin Ashi Renko Kagi Line Break Point & Figure The skill distinguishes between: Synthetic Chart Used for Signal Generation and Synthetic Chart Prices Used as Execution Prices because synthetic prices may not correspond directly to executable market prices. The skill audits intrabar ambiguity. Examples include: Entry and Stop Touched in the Same Bar Entry and Target Touched in the Same Bar Stop and Target Both Touched in the Same Bar Backtest Assuming the Favorable Price Sequence When the intrabar path is unknown, the skill can recommend: Lower-Timeframe Validation Conservative Fill Assumptions Bar Magnifier / Equivalent Validation Where Available Alternative Execution Tests The skill audits execution realism. It reviews: Market Orders Limit Orders Stop Orders Stop-Limit Orders Same-Bar Orders Next-Bar Orders Bar-Close Orders Order Cancellation Order Replacement Partial Exits Trade Reversals The skill evaluates whether historical fills are realistically obtainable. Transaction costs receive dedicated analysis. Commission can be reviewed as: Per Order Per Contract Percentage Per Side Round Trip The skill flags: Missing Commission Wrong Commission Unit Unrealistically Small Commission One-Sided Costs Cost Assumptions Inconsistent with Trading Frequency Slippage is evaluated relative to: Instrument Timeframe Liquidity Volatility Order Type Session News Sensitivity Expected Size Recommended stress scenarios can include: Base Slippage 1.5× Slippage 2× Slippage 3× Slippage Spread is evaluated when relevant, especially for: Forex Crypto Scalping Systems Thinly Traded Markets Short-Timeframe Strategies The skill never assumes unlimited liquidity. When trade size and capacity cannot be evaluated, it explicitly states: Capacity Not Evaluated The skill supports market-specific validation. For futures, it can consider: Continuous Contract Construction Roll Logic Back Adjustment Tick Size Tick Value Contract Liquidity Per-Side Commission Slippage in Ticks Overnight Sessions Margin Assumptions For crypto, it can consider: 24/7 Trading Exchange Differences Maker/Taker Fees Funding for Perpetual Contracts Liquidation Risk Exchange Outages Liquidity Changes Spread Variability For forex, it can consider: Spread Broker Feed Session Effects Rollover Bid/Ask Differences Weekend Gaps News Slippage For equities, it can consider: Splits Dividends Delistings Survivorship Bias Borrow Availability Short-Sale Constraints Opening Gaps Liquidity The skill audits position sizing. It can distinguish between: Fixed Quantity Fixed Cash Percent of Equity Volatility-Based Sizing Compounding Leveraged Exposure It can flag: Unrealistic Size Growth Hidden Path Dependence Unconstrained Leverage Exposure That Expands During Favorable Historical Periods Pyramiding can be evaluated for: Maximum Concurrent Entries Average Exposure Worst-Case Exposure Stop Handling Margin Implications The skill evaluates stop-loss logic. It checks: Stop Distance Stop Price Calculation Timing Gap Behavior Intrabar Behavior Volatility Dependence Tick-Size Constraints Take-profit logic can be reviewed for: Target Distance Limit Fill Assumptions Same-Bar Behavior Partial Exits Unrealistically Favorable Price Touches Trailing stops can be reviewed for: Activation Trail Distance Update Timing Intrabar Dependency Historical vs Realtime Differences The skill performs data-integrity review. It can check: Correct Symbol Continuous vs Individual Futures Contracts Contract Rolls Adjusted vs Unadjusted Data Corporate Actions Session Templates Timezone Missing Bars Illiquid Periods Historical Depth Symbol Changes Data-Vendor Differences Survivorship Bias The skill reviews natural-language strategies before backtesting. Ambiguous language such as: Strong Trend Good Momentum Near Support Clear Reversal High Volume must be converted into measurable definitions. The skill can produce an exact test specification containing: Entry Exit Stop Target Position Size Session Pyramiding Reversal Cooldown Order Timing Invalidation Data Source This prevents discretionary hindsight from being disguised as systematic logic. The skill gives major importance to in-sample versus out-of-sample testing. In-Sample data includes any history used to: Create Rules Choose Indicators Choose Filters Tune Parameters Choose Stops Choose Targets Choose Markets Choose Timeframes Out-of-Sample data should not be used during development. The skill identifies OOS contamination. For example: Test OOS → Change Filter → Retest Same OOS → Change Stop → Retest Same OOS This means the supposed OOS period has become part of strategy development. The skill can recommend: Development Data Validation Data Final Untouched Holdout It can compare IS and OOS using: Trade Count Net Profit Profit Factor Expectancy Sharpe Sortino Maximum Drawdown Win Rate Payoff Ratio Exposure Loss Streaks The skill does not require identical IS and OOS results. Some degradation is expected. It instead evaluates whether degradation is plausible or catastrophic. The skill can audit walk-forward testing. Possible workflow: Train / Calibrate → Freeze → Test Next Unseen Segment → Roll Forward → Repeat It reviews: Training-Window Length Testing-Window Length Overlap Leakage Reoptimization Frequency Parameter Drift Out-of-Sample Aggregation Regime Coverage It supports: Anchored Walk-Forward Rolling Walk-Forward without assuming one is universally superior. The skill explicitly analyzes selection bias. Potential causes include: Many Indicators Tested Many Assets Tested Many Timeframes Tested Many Stop/Target Combinations Many Filters Many Entry Rules Thousands of Parameter Combinations Only the Winning Strategy Reported The skill asks: How many materially different variants were tested before this one? When unknown, it states: Selection-Bias Exposure Cannot Be Quantified The skill evaluates multiple-testing risk. It considers: Search-Space Size Number of Configurations Optimization Objective Manual Experimentation Stopping Rule Number of Markets Number of Timeframes The skill evaluates research degrees of freedom. Potential dimensions include: Indicator Type Indicator Length Threshold Session Timeframe Stop Target Filter Market Entry Timing Exit Timing Confirmation Logic More flexible systems require stronger independent validation. The skill performs parameter-sensitivity analysis. Each parameter can be reviewed using: Selected Value Nearby Values Performance Neighborhood Parameter Surface Shape Possible parameter-surface classifications: Broad Plateau Gradual Region Cliff Isolated Spike A broad stable region is generally stronger evidence than one exceptional parameter surrounded by weak results. The skill can recommend perturbations such as: Nearby Integers ±5% ±10% Nearby Thresholds Nearby Stop Distances Nearby Target Distances For interacting parameters, it can evaluate joint sensitivity. A stable ridge or plateau is generally more credible than a single isolated optimum. The skill assesses trade-count adequacy. It does not claim that any universal trade count guarantees robustness. A practical evidence heuristic can classify: Under 30 Trades: Very Weak Evidence 30–99 Trades: Weak Evidence 100–299 Trades: Moderate Evidence 300+ Trades: Stronger Statistical Base These are context-dependent guidelines, not guarantees. The skill also evaluates whether trades are actually independent. Trade dependence can arise from: Pyramiding Multiple Entries During One Trend Overlapping Positions One Event Generating Many Trades Highly Correlated Instruments Clustered Signals The nominal trade count may therefore exaggerate effective evidence. The skill evaluates Win Rate. Formula: Winning Trades / Total Trades Win Rate is never interpreted in isolation. The skill evaluates Payoff Ratio. Formula: Average Winner / Absolute Average Loser Win rate and payoff should be analyzed together. The skill evaluates expectancy. Formula: (Win Rate × Average Winner) - (Loss Rate × Absolute Average Loser) Expectancy can be expressed in: Currency Points Ticks R-Multiples Percentage depending on the available data. The skill evaluates Profit Factor. Formula: Gross Profit / Absolute Gross Loss Profit Factor is evaluated for: Magnitude Period Stability OOS Stability Cost Sensitivity Dependence on Large Winners A high Profit Factor does not automatically imply robustness. The skill evaluates profit concentration. Possible checks include: Largest Winner Contribution Top 3 Winners Contribution Top 5 Winners Contribution Top 10 Winners Contribution The skill can recommend winner-removal stress tests. Example: Remove Largest Winner Remove Top 3 Winners Remove Top 5 Winners If the entire edge disappears after removing a small number of trades, confidence is reduced. The skill evaluates loss tails. It can inspect: Largest Loss Top Losses Gap Losses Stop Overruns Loss Clusters Tail Asymmetry The skill evaluates Sharpe Ratio. It recognizes limitations including: Sparse Trades Return-Frequency Sensitivity Serial Correlation Skewed Returns Smoothing Platform-Specific Calculation Methods The skill evaluates Sortino Ratio. It recognizes that downside-focused ratios still depend heavily on: Sample Size Return Construction Outliers Dependence Calculation Method When platform methodology is unknown, the skill labels it explicitly as unverified. The skill performs comprehensive drawdown analysis. Potential dimensions include: Absolute Drawdown Percentage Drawdown Peak-to-Trough Drawdown Closed-Equity Drawdown Open-Equity Drawdown Drawdown Duration Recovery Time Longest Underwater Period A strategy with moderate drawdown but extremely long recovery may still be operationally difficult. The skill evaluates drawdown concentration across: Trend Regimes Range Regimes High Volatility Low Volatility Gaps News Structural Transitions The skill evaluates maximum consecutive losses. It can review: Maximum Losing Streak Frequency of Long Streaks Loss Clustering Regime Association The skill does not convert these results into personalized financial recommendations. The equity curve is reviewed for: Smoothness Large Step Changes One-Time Jumps Long Stagnation Structural Breaks Sudden Collapse Suspicious Monotonicity A smooth equity curve is not automatically considered strong evidence. The skill can review trade-return distributions. Potential statistics include: Mean Median Standard Deviation Skew Quantiles Outliers Tail Losses A positive mean combined with a negative median can reveal dependence on rare large winners. The skill evaluates market exposure. Potential dimensions include: Percent Time in Market Long Exposure Short Exposure Simultaneous Positions Leverage Where appropriate, performance can be compared with: Buy-and-Hold Cash Simple Trend Rule Simpler Strategy Variant The skill can challenge excessive strategy complexity. If a complex strategy produces little OOS improvement over a simple rule, simplification may improve robustness. The skill performs regime analysis. Potential regimes include: Bull Trend Bear Trend Sideways High Volatility Low Volatility Crisis Recovery Liquid Periods Illiquid Periods Regime classification should be defined independently where possible. The skill does not create favorable regime definitions retrospectively merely to rescue poor performance. A regime matrix can contain: Regime Trades Expectancy Profit Factor Drawdown Status Possible statuses: Stable Acceptable Weak Failing Insufficient Sample The skill performs period robustness analysis. History may be split by: Year Quarter Rolling Window Equal-Length Block Market Cycle A strategy that earns almost all of its profit during one short historical segment receives additional scrutiny. The skill supports rolling performance review. Possible windows include: 3 Months 6 Months 12 Months depending on strategy frequency and data length. The skill evaluates: Stability Deterioration Recovery Structural Breaks The skill supports cross-market robustness tests where economic transferability is logical. Potential examples include: Multiple Equity Indices Related Futures Markets Several Liquid FX Pairs Several Large-Cap Crypto Markets Cross-market performance is supporting evidence, not a universal requirement. The skill supports cross-timeframe testing when the underlying logic should reasonably scale. It recognizes that market microstructure can change materially between timeframes. The skill performs cost-stress testing. Potential scenarios: Base Costs Moderately Higher Costs Severe Costs The question is whether the strategy remains economically coherent when execution becomes less favorable. The skill performs delay stress testing where appropriate. Examples: One-Bar-Later Entry Less Favorable Entry Less Favorable Exit A strategy that collapses after tiny timing degradation may be fragile. Price perturbation can be performed using: Ticks Spread Fraction of ATR depending on the market. The skill performs rule perturbation. Examples: Slightly Different Threshold Small Parameter Change Slight Timing Change Removal of One Filter If every exact historical rule is required for profitability, overfitting risk increases. The skill supports filter ablation. Each filter can be removed individually to determine whether it adds stable value or merely captures historical exceptions. Feature ablation can compare: Base Strategy Base + Filter A Base + Filter B Base + Filter C The skill supports Monte Carlo analysis when trade-level data is available. Possible tests include: Trade-Order Reshuffling Bootstrap Resampling Slippage Perturbation Missed Trades Execution Noise Parameter Perturbation Possible outputs include: Median Terminal Result Drawdown Distribution High-Percentile Drawdown Probability of Negative Result Under the Simulation Assumptions Loss-Streak Distribution Monte Carlo results are treated as conditional simulations, not certainty. The skill can recommend bootstrap analysis. Where trade dependence exists, it recognizes that independent resampling may be inappropriate and that block-based methods may be required. The skill identifies data snooping. Repeated exposure to the same historical data during strategy development means the strategy has learned from that period, even when tuning was performed manually rather than through automated optimization. The skill identifies survivorship bias. For multi-stock studies, it can ask whether: Delisted Companies Failed Companies Historical Constituents were included. The skill identifies execution leakage. A strategy may not explicitly use future data but may still create impossible historical results through unrealistic fill sequencing. The skill audits trade lists. Potential fields include: Entry Timestamp Exit Timestamp Duration Direction Position Size P&L MAE MFE Entry Reason Exit Reason Order Type MAE and MFE can help evaluate: Stop Efficiency Target Efficiency Exit Quality Tail Exposure The skill compares backtest and live or paper results when available. Potential comparisons include: Signal Count Fill Price Slippage Missed Trades Win Rate Expectancy Average Trade Drawdown Gap Behavior Latency The skill supports forward-test discipline. A recommended freeze protocol is: Freeze Strategy Logic Freeze Parameters Record Version Record Assumptions Test on Unseen Data Do Not Tune During the Test Record Deviations Separately The skill can create strategy versioning records containing: Version Date Logic Changes Parameter Changes Reason Historical Data Seen Before the Change This provides an audit trail for retrospective tuning. The skill uses a strict robustness-evidence hierarchy. Stronger evidence generally includes: Valid Test Realistic Costs Adequate Sample True Unseen Data Stable Nearby Parameters Multiple Periods Multiple Regimes Stress-Test Survival Credible Forward Behavior Weaker evidence typically includes: One Optimized Backtest One Market One Period Unknown Selection History Favorable Execution Narrow Parameter Optimum Small Sample The final result is classified into exactly one category. ROBUST Used only when: No Critical Test-Integrity Failure Is Detected Execution Assumptions Are Plausible Meaningful OOS Evidence Exists OOS Behavior Remains Economically Coherent Sample Size Is Reasonable Parameters Show Stability Performance Is Not Critically Concentrated Drawdown Is Understood Multiple Periods Are Acceptable Regime Behavior Is Understood Stress Tests Do Not Immediately Destroy the Edge No Major Unresolved Bias Exists Required interpretation: ROBUST — The available evidence shows comparatively strong historical robustness under the tested assumptions. This does not guarantee future profitability. PROMISING BUT UNPROVEN Used when: The Test Appears Valid A Potential Edge Exists Evidence Remains Incomplete Possible causes: Limited OOS Moderate Trade Count No Walk-Forward No Cost Stress Limited Regime Coverage No Frozen Forward Test Incomplete Parameter Analysis OVERFIT RISK Used when the test is technically interpretable but historical fragility is material. Possible causes: Narrow Parameter Optimum Major IS/OOS Collapse Large Search Space Profit Concentration One Period Producing Most Profit Unexplained Regime Dependence Cost Fragility Complex Rule Stack Repeated OOS Reuse Instability Under Small Changes INVALID TEST Used when a fundamental integrity issue prevents credible interpretation. Examples: Look-Ahead Bias Confirmed Repainting Future Leakage Impossible Execution Materially Invalid Same-Bar Fill Assumptions Synthetic Prices Used Incorrectly Corrupted or Misaligned Data Broken Logic Reported Results Not Matching the Tested Strategy Classification precedence is: INVALID TEST → OVERFIT RISK → PROMISING BUT UNPROVEN → ROBUST A critical validity failure takes precedence over every profitability metric. Every final audit can also include: Classification Confidence: High Medium Low The skill can create an optional 0–5 scorecard covering: Test Integrity Execution Realism Sample Adequacy OOS Quality Selection-Bias Control Parameter Stability Drawdown Quality Regime Robustness Period Robustness Statistical Robustness Reproducibility This score is never presented as a probability of future profitability. The skill can produce: Executive Verdict Final Classification Classification Confidence Strategy Summary Test Validity Review Pine Script Integrity Review Data Integrity Review Execution Realism Review Sample Adequacy IS vs OOS Analysis Selection-Bias Analysis Parameter-Sensitivity Analysis Core Metric Review Trade-Distribution Review Drawdown Analysis Consecutive-Loss Analysis Regime Analysis Period Robustness Cross-Market Robustness Stress-Test Results Red-Flag Register Unknowns Register Required Retests Final Conclusion The skill is built for traders who want a rigorous second opinion on backtest credibility rather than another system that simply celebrates a high Profit Factor or attractive equity curve.