More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Rulebook Writer + Linter: rules two judges read the same way

    1

    Turn inconsistent LLM rubrics into precise, linted rulebooks with checkable statements and stable severities.

    $9

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Rulebook Writer + Linter: rules two judges read the same way

    Rulebook Writer + Linter: rules two judges read the same way

    Example session with this skill installed

    I have a customer support chatbot that sometimes promises refunds it shouldn't. I need a rulebook for a judge to review its transcripts. Here are 3 examples of it giving money away wrongly.

    • Read your context and instructions
    • Compiled the rulebook writer

    I have analyzed the failure cases and initialized a new rulebook. I have converted your refund failures into pointable rules with specific boundaries and centralized the refund limits in the Definitions section.

    Support Transcript Rulebook

    
    # Rulebook: Support Transcript Review
    **Scope**: Reviews support chatbot transcripts to ensure financial promises align with company policy.
    
    ## Definitions
    - **REFUND_LIMIT**: $50.00 USD. Any amount above this requires human escalation.
    - **APPROVED_REASONS**: "Damaged Item", "Late Delivery (>3 days)", "Incorrect Item".
    - **ESCALATION_PHRASE**: "I will need to transfer you to a supervisor to process this request."
    
    ## Rules
    - **R001 (Blocker)**: Every refund offered must be for an amount less than or equal to the REFUND_LIMIT. 
      - *Pointable*: The specific dollar amount mentioned in the chat.
    - **R002 (Major)**: Every refund offered must be for a reason listed in APPROVED_REASONS.
      - *Pointable*: The sentence stating the reason for the refund.
    - **R003 (Major)**: The agent must use the ESCALATION_PHRASE if the user requests an amount exceeding REFUND_LIMIT.
      - *Pointable*: The response immediately following the user's request.
    
    ## Boundaries
    - **R001 vs R003**: R001 flags the act of promising too much money; R003 flags the failure to follow the escalation protocol. If both occur, flag both.
    
    ## Lint Status
    - **ERROR**: None.
    - **WARNING**: W102 (Definitions) - "Late Delivery" definition uses a relative time (>3 days); ensure the judge has access to the order date context.
    

    Next steps

    1. Run python scripts/rulebook.py lint rules.md --strict to confirm the W102 warning is acceptable for your pipeline.
    2. Provide two "golden" transcripts that followed policy so I can verify the rules don't trigger false positives.
    3. Run python scripts/rulebook.py export rules.md --format prompt to get the text for your LLM judge system message.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most “unreliable” LLM judges apply rules that two people would read differently. Vague words like “appropriate”. Limits kept in a policy the judge never sees. Two rules that both describe one defect, so a correct finding gets scored as wrong. This skill guides your agent from past failures and examples to checkable rules, with severities set by consequence, definitions written in, and boundaries between rules. Then it lints the result.

    Fourteen checks catch bad severities, duplicate or reused IDs, rules that depend on a file the judge can’t see, vague wording, amounts without numbers, likely overlaps with no stated boundary, and unscoped wording rules. Export it as judge-ready text or for the Reviewer Calibration Kit, and diff versions safely.

    Python 3.10+, nothing to install. Includes a writing guide, a lint-code reference, a template, a before-and-after example and 29 tests.

    Inconsistent judges and ambiguous review criteria lead to unreliable LLM evaluations. When two judges read the same rubric and reach different verdicts, the problem is usually vague wording, overlapping rules, or missing definitions. This skill eliminates that variance by turning past failures and examples into a structured, checkable rulebook.

    What it does

    • Rule drafting converts user examples of bad outputs into precise, checkable statements with assigned severities.
    • Automated linting identifies vague wording, duplicate IDs, bad severities, and overlapping logic using a dedicated Python CLI.
    • Definition scoping extracts limits, lists, and special terms into a centralized definitions block for the judge to reference.
    • Boundary mapping explicitly defines where one rule ends and another begins to prevent double-flagging or confusion.
    • Multi-format export generates judge-ready prompt text or JSON suites compatible with calibration kits.

    How it works

    1. Collect inputs: The agent asks for the review object, past failure examples, and two "golden" good examples to set the baseline.
    2. Initialize rulebook: The rulebook.py new command scaffolds the file with stable IDs and standard sections.
    3. Draft and scope: The agent writes the rules, definitions, and boundaries based on the references/writing-rules.md standard.
    4. Lint and fix: Run rulebook.py lint to catch structural errors or logical overlaps before the judge ever sees the rubric.
    5. Calibrate and export: The final rulebook is exported as a prompt or a suite for integration into an evaluation pipeline.

    Frameworks & tools

    The core logic resides in a standalone Python 3.10+ script (rulebook.py) with no external dependencies. It includes templates and reference guides for rule anatomy and lint codes.

    Why this beats prompting it yourself

    Prompting a judge with a list of "things to look for" results in drift. This skill enforces a strict schema where every defect is pointable and every value is defined. It prevents the two most common judge failures: flagging valid content due to broad rules and missing defects because the judge lacks a specific limit.

    Use cases

    • Drafting a grading rubric for student essays to ensure consistent scoring across TA runs.
    • Building a content-moderation checklist for an LLM that flags specific policy violations without false positives.
    • Creating a code-review rulebook that enforces styling and security standards with clear major/minor severities.
    • Generating a quality-assurance suite for customer support agents that identifies prohibited phrasing or incorrect technical info.

    Known limitations

    The Python script requires a local 3.10+ environment to run commands. The skill does not automate the final calibration against a test set but prepares the files for that process.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.

    Requires Python 3.10+. Does not perform the actual LLM judge execution. Clean lint results do not guarantee rule accuracy, only structural integrity.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 11 days ago

    Software
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed11 days ago

    What's inside

    Frequently Asked Questions

    This skill replaces vague, bulleted rubrics with structured rulebooks that use a Python-based linting system to eliminate overlapping rules, ambiguous definitions, and severity drift that typically lead to inconsistent LLM judge scores.

    The purchase includes the rulebook.py CLI script, a comprehensive library of rule-writing reference guides, standard templates for rule anatomy, and the AI agent instructions needed to orchestrate the drafting process.

    The core logic is a standalone Python 3.10+ script with zero external dependencies, making it compatible with any local environment or CI/CD pipeline that can execute Python commands.

    While the agent drafts the rules, you will need to provide at least two 'golden' (good) examples and a few examples of past failures to help the agent map the specific boundaries and severities for your use case.

    The skill is designed to produce judge-ready text or JSON suites that work with any major LLM (like GPT-4, Claude, or Llama) and can be easily integrated into existing evaluation frameworks or calibration kits.

    The automated linter flags structural errors and logical overlaps, but it does not perform the final calibration against your test set; it prepares the files so that your human-in-the-loop calibration is faster and more accurate.

    New in Agents & Orchestration