Judge Calibration Bundle: write the rules, then prove your judge applies them
Two skills that together make an LLM judge you can rely on. The Rulebook Writer + Linter turns past failures into checkable rules, with severities, written-in definitions and clear boundaries between rules, and lints them before any judge runs. The Reviewer Calibration Kit then proves the judge applying those rules catches planted defects and invents none. It plants defects with near-identical clean twins, keeps the answers hidden, runs the judge several times, and scores recall, false findings and verdict agreement.
One path from “my judge seems inconsistent” to a scored report you can show. Python 3.10+, nothing to install, worked examples and tests in both. Save $5 against buying them separately.
Works with any agent that signs in to Agensi, such as Claude, ChatGPT, Claude Code, Cursor and Codex CLI.
One payment, lifetime access. 2 skills unlock instantly in your library.
What's included
2 skillsTurn inconsistent LLM rubrics into precise, linted rulebooks with checkable statements and stable severities.
Build and score evaluation suites to measure recall, false positives, and verdict accuracy for LLM-as-a-judge agents.
See it in action
Rulebook Writer + Linter: rules two judges read the same way
Example session with this skill installed
I have a customer support chatbot that sometimes promises refunds it shouldn't. I need a rulebook for a judge to review its transcripts. Here are 3 examples of it giving money away wrongly.
- Read your context and instructions
- Compiled the rulebook writer
I have analyzed the failure cases and initialized a new rulebook. I have converted your refund failures into pointable rules with specific boundaries and centralized the refund limits in the Definitions section.
Support Transcript Rulebook
# Rulebook: Support Transcript Review
**Scope**: Reviews support chatbot transcripts to ensure financial promises align with company policy.
## Definitions
- **REFUND_LIMIT**: $50.00 USD. Any amount above this requires human escalation.
- **APPROVED_REASONS**: "Damaged Item", "Late Delivery (>3 days)", "Incorrect Item".
- **ESCALATION_PHRASE**: "I will need to transfer you to a supervisor to process this request."
## Rules
- **R001 (Blocker)**: Every refund offered must be for an amount less than or equal to the REFUND_LIMIT.
- *Pointable*: The specific dollar amount mentioned in the chat.
- **R002 (Major)**: Every refund offered must be for a reason listed in APPROVED_REASONS.
- *Pointable*: The sentence stating the reason for the refund.
- **R003 (Major)**: The agent must use the ESCALATION_PHRASE if the user requests an amount exceeding REFUND_LIMIT.
- *Pointable*: The response immediately following the user's request.
## Boundaries
- **R001 vs R003**: R001 flags the act of promising too much money; R003 flags the failure to follow the escalation protocol. If both occur, flag both.
## Lint Status
- **ERROR**: None.
- **WARNING**: W102 (Definitions) - "Late Delivery" definition uses a relative time (>3 days); ensure the judge has access to the order date context.
Next steps
- Run
python scripts/rulebook.py lint rules.md --strictto confirm the W102 warning is acceptable for your pipeline. - Provide two "golden" transcripts that followed policy so I can verify the rules don't trigger false positives.
- Run
python scripts/rulebook.py export rules.md --format promptto get the text for your LLM judge system message.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads these skills and uses them on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every skill in this bundle goes through our trust checks.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.