BUNDLE Security scanned2 skills

    Judge Calibration Bundle: write the rules, then prove your judge applies them

    Two skills that together make an LLM judge you can rely on. The Rulebook Writer + Linter turns past failures into checkable rules, with severities, written-in definitions and clear boundaries between rules, and lints them before any judge runs. The Reviewer Calibration Kit then proves the judge applying those rules catches planted defects and invents none. It plants defects with near-identical clean twins, keeps the answers hidden, runs the judge several times, and scores recall, false findings and verdict agreement.

    One path from “my judge seems inconsistent” to a scored report you can show. Python 3.10+, nothing to install, worked examples and tests in both. Save $5 against buying them separately.

    Works with any agent that signs in to Agensi, such as Claude, ChatGPT, Claude Code, Cursor and Codex CLI.

    Samith Weerasuriya
    Created by
    Samith Weerasuriya
    $19$24
    Save 21% · $5

    One payment, lifetime access. 2 skills unlock instantly in your library.

    Money-back guarantee

    Instant unlock in your library

    Free updates from the creator

    What's included

    2 skills

    See it in action

    Rulebook Writer + Linter: rules two judges read the same way

    Rulebook Writer + Linter: rules two judges read the same way

    Example session with this skill installed

    I have a customer support chatbot that sometimes promises refunds it shouldn't. I need a rulebook for a judge to review its transcripts. Here are 3 examples of it giving money away wrongly.

    • Read your context and instructions
    • Compiled the rulebook writer

    I have analyzed the failure cases and initialized a new rulebook. I have converted your refund failures into pointable rules with specific boundaries and centralized the refund limits in the Definitions section.

    Support Transcript Rulebook

    
    # Rulebook: Support Transcript Review
    **Scope**: Reviews support chatbot transcripts to ensure financial promises align with company policy.
    
    ## Definitions
    - **REFUND_LIMIT**: $50.00 USD. Any amount above this requires human escalation.
    - **APPROVED_REASONS**: "Damaged Item", "Late Delivery (>3 days)", "Incorrect Item".
    - **ESCALATION_PHRASE**: "I will need to transfer you to a supervisor to process this request."
    
    ## Rules
    - **R001 (Blocker)**: Every refund offered must be for an amount less than or equal to the REFUND_LIMIT. 
      - *Pointable*: The specific dollar amount mentioned in the chat.
    - **R002 (Major)**: Every refund offered must be for a reason listed in APPROVED_REASONS.
      - *Pointable*: The sentence stating the reason for the refund.
    - **R003 (Major)**: The agent must use the ESCALATION_PHRASE if the user requests an amount exceeding REFUND_LIMIT.
      - *Pointable*: The response immediately following the user's request.
    
    ## Boundaries
    - **R001 vs R003**: R001 flags the act of promising too much money; R003 flags the failure to follow the escalation protocol. If both occur, flag both.
    
    ## Lint Status
    - **ERROR**: None.
    - **WARNING**: W102 (Definitions) - "Late Delivery" definition uses a relative time (>3 days); ensure the judge has access to the order date context.
    

    Next steps

    1. Run python scripts/rulebook.py lint rules.md --strict to confirm the W102 warning is acceptable for your pipeline.
    2. Provide two "golden" transcripts that followed policy so I can verify the rules don't trigger false positives.
    3. Run python scripts/rulebook.py export rules.md --format prompt to get the text for your LLM judge system message.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads these skills and uses them on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every skill in this bundle goes through our trust checks.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Frequently asked questions

    Yes — every skill in Judge Calibration Bundle: write the rules, then prove your judge applies them is also sold on its own product page. The bundle just packages them together at a discount (save 21% vs buying each one separately).

    Yes. When you own the bundle your connected agent can load the latest version of any included skill. Updates the creator publishes after your purchase are included at no extra cost.

    All skills in this bundle use the open SKILL.md standard and work with Claude Code, Cursor, Codex CLI, Gemini CLI, GitHub Copilot, Windsurf. The compatibility list on this page is computed from the included skills.

    The bundle price is set 21% below the combined retail price of the 2 paid skills included. You pay once and unlock every skill in your library instantly.

    Bundles are covered by the same money-back guarantee as individual skills. Doesn't do what its page promises? You get your money back.