experiment reproducibility auditor

The problem

Machine learning experiments often fail to replicate because of missing lineage between code, data, and environment. Teams struggle to verify if a specific model artifact actually matches a specific MLflow run or Git commit.

What it does

• Maps Git commits, MLflow metadata, and DVC hashes into a single evidence chain.
• Identifies missing links in the reproduction pipeline before scoring results.
• Flags statements as Verified, Inferred, or Assumed to prevent false confidence.
• Generates a structured reproducibility audit with implementation steps for recovery.

Frameworks & tools

MLflow, DVC, Git, and various experiment manifests or logs.

Why this beats prompting it yourself

Standard LLM prompts often assume tool presence implies configuration maturity. This skill enforces a strict evidence contract that refuses to infer capability from tool names and explicitly tracks unverified assumptions through a specialized suite trace.

Use cases

• Auditing a model run to ensure it can be rebuilt from scratch.
• Identifying why a local experiment doesn't match a remote baseline.
• Verifying lineage for regulatory compliance or peer review.
• Mapping disparate metadata from multiple ML tools into one source of truth.

Known limitations

Does not debug failed training runs or select orchestrators. Does not execute deployments or handle credentials.