EvalForge — Mechanical Eval Gates for Agents

EvalForge turns operational material into eval fixtures that test real agent failure patterns, and checks agent output against them with plain pass/fail rules.

What ships in the folder:

• 12 gate contracts, G01–G12, each describing one class of agent failure (redacted tool output, code-execution reset, command timeout, empty search result, image sentinel, memory/context receipt, citation grammar, truncation disclaimer, files-loaded sentinel, thought-trace interruption, minimal-ack tool returns, destructive-action provenance).
• Two strict JSON schemas: one for eval cases and one for run intakes.
• Four dependency-free validators (eval cases, gate contracts, run intakes and phase cards), a mechanical pass/fail runner, and a score-report generator, plus a shared schema helper. Plain Python 3.8+; jsonschema is used if present, otherwise a built-in checker.
• A mechanical pass/fail runner with no LLM judge: each case is scored by regex and rule checks against the model output, exiting 0 on pass and 1 on fail.
• A score report generator and report template.
• Three worked cases with known-pass and known-fail fixtures (G02, G04, G08).
• Templates for gate contracts, phase cards, run intakes and receipts.

Worked cases with known-pass/known-fail fixtures exist for 3 of the 12 gates (G02, G04, G08); the other 9 gates ship as contracts and validators only.

This is a mechanical validation spine, not a full semantic evaluation suite.

Usage example: Run the G08 truncation-disclaimer case against this agent output and tell me whether it passes. Works with: any agent that loads SKILL.md files and can run Python 3.8+.