Training Pipeline Failure Forensics

Turn a failed training run into a ranked investigation

Training failures appear in two forms: loud crashes and silent degradation. This skill separates the symptom from the cause, timelines what changed since the last good run, and tests six failure classes against supplied evidence.

What it produces

• A loud-versus-silent failure classification
• A change-set timeline across data, code, dependencies, infrastructure, configuration, and orchestration
• Ranked hypotheses with evidence for and against each one
• A confidence label and cheapest discriminating test per hypothesis
• A minimal reproduction path for the leading hypothesis
• A prevention line that can become a pipeline gate

Failure classes covered

Data-contract breaks, leakage, environment or dependency drift, resource and numerical failures, code defects, and orchestration faults.

Best for

• Training jobs that crash with partial logs
• Runs that complete but produce worse metrics
• Teams comparing the last good run with a failed one
• Reviewers who need a minimal test before authorizing a rerun

Boundaries

The skill ranks hypotheses; it does not prove causality, execute SQL or training jobs, repair code, access infrastructure, or guarantee that the leading hypothesis is correct. Confirmation requires the named discriminating test.