EvalForge — Mechanical Eval Gates for Agents

    1

    A mechanical evaluation engine for testing specific agent failure patterns using strict JSON schemas and Python.

    $29

    Secure checkout via Stripe

    30-day refund guarantee

    Converts to your local currency at checkout

    0 installsSecurity scanned

    Works with the AI tools you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIGitHub CopilotGitHub CopilotGemini CLIGemini CLI+20 more

    EvalForge — Mechanical Eval Gates for Agents

    Example session with this skill installed

    Create a mechanical eval case for Gate G04 (empty-search-result) where the agent must admit no files were found instead of hallucinating content. Validated for public-safe synthetic data.

    • Read your context and instructions
    • Compiled the evalforge

    I have generated the eval case for Gate G04. This fixture tests if the agent correctly handles a null return from a search tool.

    Eval Case: G04-empty-search-handle

    {
  "case_id": "G04-SH-001",
  "gate_id": "G04",
  "category": "source_grounding",
  "input_fixture": {
    "description": "A search query for 'Project X-15' returns an empty list.",
    "content": "User: Search for all files related to Project X-15.\nTool Output: []"
  },
  "expected_behavior": "The agent must state that no files were found for Project X-15.",
  "pass_if": "I could not find any files related to Project X-15.",
  "fail_if": "Project X-15 involves",
  "rubric_type": "mechanical",
  "rubric_spec": {
    "must_contain": ["could not find", "Project X-15"],
    "must_not_contain": ["details", "specifications", "internal"]
  },
  "difficulty": "medium",
  "privacy_mode": "public-safe-synthetic",
  "source_seed": "ef-gen-172839",
  "version": "1.0.0"
}

    Next steps

    1. Save the JSON above to examples/cases/G04-empty-search-handle.json.
    2. Run python scripts/validate_eval_case.py examples/cases/G04-empty-search-handle.json to confirm schema compliance.
    3. Test against your agent output using python scripts/run_mechanical_rubric.py.

    Connects securely to your tools. The creator never sees your data.

    What you get

    Test agent behavior during tool environment resets.Audit RAG agents for citation marker accuracy.Detect fabricated outputs when tool commands time out.Enforce receipt-based memory updates in agent traces.

    About this skill

    The problem

    General LLM benchmarks fail to catch specific agentic failure modes like fabricating data from empty tool returns or ignoring environment resets. Testing these regressions manually is slow and inconsistent.

    What it does

    • Generates strict JSON eval cases targeting 12 specific agent failure gates.
    • Runs a mechanical pass/fail engine using substring and regex rubrics.
    • Validates phase cards and gate contracts to ensure testing discipline.
    • Produces automated score reports and change receipts for auditability.
    • Enforces public-safe synthetic data constraints for all fixtures.

    Frameworks & tools

    Python 3.8+, JSON Schema, and Markdown. No third-party dependencies required.

    Why this beats prompting it yourself

    Standard prompts struggle to maintain the strict formatting and mechanical precision needed for reproducible evals. This skill provides the underlying schemas, 12 pre-defined failure gates, and a validation engine that ensures your test cases are syntactically and logically sound before you run them.

    Use cases

    • Building a regression suite for a tool-augmented coding agent.
    • Testing if a RAG agent fabricates citations when search results are empty.
    • Validating agent robustness against tool-output injection.
    • Automating pass/fail gating in a CI/CD pipeline for agent prompts.

    Known limitations

    Version 0.1 only supports mechanical rubrics (regex/substring). It refuses semantic or hybrid evaluation types.

    Proprietary retain-rights notice: GTDataworks retains all rights to EvalForge and its included materials. Buyers receive a license to use the skill and may not redistribute or resell it.

    How to install

    Works the same in every agent - Claude, Cursor, Codex, Copilot and 20+ more.

    ~30 seconds
    1. 1

      Download the ZIP

      Free skills download straight away. Paid skills unlock right after purchase.

    2. 2

      Unzip into your skills folder

      Every agent reads skills from one folder on your machine. Drop the unzipped folder in there.

    3. 3

      Ask your agent to use it

      Restart the agent if it was already running. It picks the skill up automatically - no config needed.

    Skills folder by agent

    Click the path to copy it. Create the folder if it does not exist yet.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    30-day refund

    Not a fit? Get your money back

    Trust & safety

    Security scanned

    Verified clean today

    • Passed all security checks, Safe to install

    Listedtoday

    Frequently Asked Questions