More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Reviewer Calibration Kit

    1

    Build and score evaluation suites to measure recall, false positives, and verdict accuracy for LLM-as-a-judge agents.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Reviewer Calibration Kit

    Reviewer Calibration Kit

    Example session with this skill installed

    I have a code reviewer agent. I need to know if it's actually catching security flaws or just guessing based on common patterns. How do I start a calibration suite?

    • Read your context and instructions
    • Compiled the reviewer calibration kit

    I will initialize a new calibration suite for your code reviewer agent. This creates the necessary structure to define your security rules and author test cases.

    Reviewer Calibration Suite: Security-Audit

    The following directory structure has been scaffolded in ./security-audit/. You should now edit rulebook.md to define your security requirements and suite.json to assign severities.

    security-audit/
    ├── rulebook.md          # Define your "No hardcoded secrets" or "SQLi" rules here
    ├── suite.json           # Map Rule IDs (e.g., SEC01) to blocker/major/minor
    ├── cases/               # Folder for your test material (K01, K02...)
    │   └── .gitkeep
    ├── keys/                # Hidden ground truth for the scorer
    │   └── .gitkeep
    └── templates/
        └── case-design.md   # Map out your Planted vs. Twin strategy
    

    Next steps

    1. Define Rules: Open security-audit/rulebook.md and write 3-5 specific, measurable security rules.

    Design Cases: Use the templates/case-design.md to plan 5 planted defects (e.g., a real API key) and 5 twins (e.g., an environment variable reference).
    3.

    Lint Suite: Run python scripts/calibrate.py lint security-audit to ensure your configuration is valid before running the agent.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Reviewer agents that "seem to work" often miss critical defects or hallucinate problems that don't exist. This skill provides a systematic framework to quantify the reliability of LLM-as-a-judge agents using a rigorous calibration process.

    What it does

    • Scaffold evaluation suites by generating structured folders for rules, test cases, and hidden keys.
    • Design adversarial test cases including planted defects, near-miss clean twins, and prompt injection attempts.
    • Automate judge execution across multiple runs to measure consistency and calculate statistical performance.
    • Generate performance reports with metrics for recall, false positives, verdict accuracy, and invalid output rates.
    • Benchmark agent tiers by comparing high-performing models against weaker ones to validate suite difficulty.

    How it works

    1. Initialize the suite with calibrate.py init to define your rulebook and severity levels (blocker, major, minor).
    2. Author cases using the case-design.md template, pairing every defect with a near-identical clean twin to test for true understanding.
    3. Execute the judge via CLI with calibrate.py run, which sends cases to your agent and saves structured JSON responses.
    4. Calculate scores using calibrate.py score to generate Markdown reports comparing agent findings against hidden keys.

    Frameworks & tools

    This kit uses a standalone Python 3.10+ CLI tool (calibrate.py). It is compatible with any LLM judge accessible via shell commands (e.g., Claude Code, local scripts, or API wrappers).

    Why this beats prompting it yourself

    Evaluating a judge with another LLM just shifts the trust problem. This tool uses a deterministic, dependency-free Python script to score agents against ground-truth keys. It forces you to define rules and severities upfront, ensuring your "calibration" is a measurable audit rather than a vibes-based check.

    Use cases

    • Validating an automated PR reviewer agent before deploying it to a production repo.
    • Calibrating a document compliance grader to ensure it doesn't hallucinate legal violations.
    • Benchmarking different LLM providers to find the cheapest model that maintains 1.0 blocker recall.
    • Identifying "severity drift" where an agent correctly finds a flaw but wrongly downgrades its importance.

    Known limitations

    The scorer requires agents to output valid JSON in a specific schema. Suites with fewer than 20 cases are flagged as statistically underpowered. Requires a CLI-accessible judge.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Requires Python 3.10+. Agent must support JSON output. Suites with <20 cases trigger statistical significance warnings. No built-in LLM providers; requires user-supplied CLI command.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 11 days ago

    Software
    Python

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed11 days ago

    What's inside

    Frequently Asked Questions

    The kit includes the `calibrate.py` CLI engine, a standardized folder structure for your evaluation data, design templates for adversarial test cases, and documentation for interpreting statistical performance reports.

    It is compatible with any LLM-as-a-judge that can be executed via a shell command or Python script. As long as your agent can receive a text input and return a structured JSON response, this kit can benchmark it.

    Unlike simple prompting, this kit uses a 'twin-case' methodology—pairing every defect with a near-identical clean version—to ensure your judge isn't just guessing based on keywords or 'vibes.'

    Yes, the kit supports benchmarking multiple agents or model tiers against the same suite, allowing you to compare the recall and accuracy of different providers (e.g., Claude vs. GPT-4) side-by-side.

    To ensure statistical significance and avoid biased results, the kit requires a minimum of 20 test cases and expects the agent to output its findings in a specific JSON schema for automated scoring.

    New in LLM Ops