- Home
- Skills
- Agents & Orchestration
- Rulebook Writer + Linter: rules two judges read the same way
More screenshots
Works with the AI Agents you already use
Rulebook Writer + Linter: rules two judges read the same way
Turn inconsistent LLM rubrics into precise, linted rulebooks with checkable statements and stable severities.
$9
Rulebook Writer + Linter: rules two judges read the same way
Example session with this skill installed
I have a customer support chatbot that sometimes promises refunds it shouldn't. I need a rulebook for a judge to review its transcripts. Here are 3 examples of it giving money away wrongly.
- Read your context and instructions
- Compiled the rulebook writer
I have analyzed the failure cases and initialized a new rulebook. I have converted your refund failures into pointable rules with specific boundaries and centralized the refund limits in the Definitions section.
Support Transcript Rulebook
# Rulebook: Support Transcript Review
**Scope**: Reviews support chatbot transcripts to ensure financial promises align with company policy.
## Definitions
- **REFUND_LIMIT**: $50.00 USD. Any amount above this requires human escalation.
- **APPROVED_REASONS**: "Damaged Item", "Late Delivery (>3 days)", "Incorrect Item".
- **ESCALATION_PHRASE**: "I will need to transfer you to a supervisor to process this request."
## Rules
- **R001 (Blocker)**: Every refund offered must be for an amount less than or equal to the REFUND_LIMIT.
- *Pointable*: The specific dollar amount mentioned in the chat.
- **R002 (Major)**: Every refund offered must be for a reason listed in APPROVED_REASONS.
- *Pointable*: The sentence stating the reason for the refund.
- **R003 (Major)**: The agent must use the ESCALATION_PHRASE if the user requests an amount exceeding REFUND_LIMIT.
- *Pointable*: The response immediately following the user's request.
## Boundaries
- **R001 vs R003**: R001 flags the act of promising too much money; R003 flags the failure to follow the escalation protocol. If both occur, flag both.
## Lint Status
- **ERROR**: None.
- **WARNING**: W102 (Definitions) - "Late Delivery" definition uses a relative time (>3 days); ensure the judge has access to the order date context.
Next steps
- Run
python scripts/rulebook.py lint rules.md --strictto confirm the W102 warning is acceptable for your pipeline. - Provide two "golden" transcripts that followed policy so I can verify the rules don't trigger false positives.
- Run
python scripts/rulebook.py export rules.md --format promptto get the text for your LLM judge system message.
Connects securely to your tools. The creator never sees your data.
About this skill
Most “unreliable” LLM judges apply rules that two people would read differently. Vague words like “appropriate”. Limits kept in a policy the judge never sees. Two rules that both describe one defect, so a correct finding gets scored as wrong. This skill guides your agent from past failures and examples to checkable rules, with severities set by consequence, definitions written in, and boundaries between rules. Then it lints the result.
Fourteen checks catch bad severities, duplicate or reused IDs, rules that depend on a file the judge can’t see, vague wording, amounts without numbers, likely overlaps with no stated boundary, and unscoped wording rules. Export it as judge-ready text or for the Reviewer Calibration Kit, and diff versions safely.
Python 3.10+, nothing to install. Includes a writing guide, a lint-code reference, a template, a before-and-after example and 29 tests.
Inconsistent judges and ambiguous review criteria lead to unreliable LLM evaluations. When two judges read the same rubric and reach different verdicts, the problem is usually vague wording, overlapping rules, or missing definitions. This skill eliminates that variance by turning past failures and examples into a structured, checkable rulebook.
What it does
- Rule drafting converts user examples of bad outputs into precise, checkable statements with assigned severities.
- Automated linting identifies vague wording, duplicate IDs, bad severities, and overlapping logic using a dedicated Python CLI.
- Definition scoping extracts limits, lists, and special terms into a centralized definitions block for the judge to reference.
- Boundary mapping explicitly defines where one rule ends and another begins to prevent double-flagging or confusion.
- Multi-format export generates judge-ready prompt text or JSON suites compatible with calibration kits.
How it works
- Collect inputs: The agent asks for the review object, past failure examples, and two "golden" good examples to set the baseline.
- Initialize rulebook: The
rulebook.py newcommand scaffolds the file with stable IDs and standard sections. - Draft and scope: The agent writes the rules, definitions, and boundaries based on the
references/writing-rules.mdstandard. - Lint and fix: Run
rulebook.py lintto catch structural errors or logical overlaps before the judge ever sees the rubric. - Calibrate and export: The final rulebook is exported as a prompt or a suite for integration into an evaluation pipeline.
Frameworks & tools
The core logic resides in a standalone Python 3.10+ script (rulebook.py) with no external dependencies. It includes templates and reference guides for rule anatomy and lint codes.
Why this beats prompting it yourself
Prompting a judge with a list of "things to look for" results in drift. This skill enforces a strict schema where every defect is pointable and every value is defined. It prevents the two most common judge failures: flagging valid content due to broad rules and missing defects because the judge lacks a specific limit.
Use cases
- Drafting a grading rubric for student essays to ensure consistent scoring across TA runs.
- Building a content-moderation checklist for an LLM that flags specific policy violations without false positives.
- Creating a code-review rulebook that enforces styling and security standards with clear major/minor severities.
- Generating a quality-assurance suite for customer support agents that identifies prohibited phrasing or incorrect technical info.
Known limitations
The Python script requires a local 3.10+ environment to run commands. The skill does not automate the final calibration against a test set but prepares the files for that process.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.
Requires Python 3.10+. Does not perform the actual LLM judge execution. Clean lint results do not guarantee rule accuracy, only structural integrity.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 11 days ago
- Software
Python