More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Judge Sanity Check — Is Your LLM Judge Stable?

    1

    Reads your paired verdicts and reports chance-corrected agreement, position flips, verbosity bias, self-preference and sample size — before a judge score gates a release.

    Free

    Security scanned
    Judge Sanity Check — Is Your LLM Judge Stable?

    Judge Sanity Check — Is Your LLM Judge Stable?

    Example session with this skill installed

    python3 scripts/judge_check.py judgments.jsonl --fail-on high

    {"item_id":"i04","pair_id":"p04","order":"AB","judge_choice":"A","human_choice":"A",
    "len_a":240,"len_b":120,"judge_model":"judge-x","a_model":"model-a","b_model":"model-b"}

    • Read your context and instructions
    • Compiled the judge sanity check

    judge-sanity-check 80 judgments 80 labelled
    agreement 38.8% kappa -0.065 (CI -0.231..0.111)
    position flips 23.3% slot-A 52.5% verbosity 58.3% (n=12)

    HIGH J2 chance-corrected agreement Cohen's kappa -0.07 is barely above chance
    HIGH J3 position bias 23.3% of pairs changed verdict when swapped
    HIGH J6 self-preference Judge picks own output 75% vs human 50%

    FAIL findings at or above 'high'

    Connects securely to your tools. The creator never sees your data.

    About this skill

    An LLM judge can give you a clean number every run and still be measuring the order you showed the candidates. This reads the verdicts you already have and tells you whether the score is stable enough to gate anything.

    It computes chance-corrected agreement with a bootstrap confidence interval, not just raw agreement. It compares the same pair shown in both orders and reports how often the verdict flips. It checks whether the first slot wins more than half the decided pairs, whether the longer answer wins pairs humans called equal, and whether the judge favours its own model's output against the human rate. It reports the tie rate, the label distribution and the number of pairs behind every number.

    • Cohen's kappa with a 95% bootstrap interval, plus the raw agreement and marginals
    • Position flips and first-slot rate from pairs with both orders
    • Verbosity bias measured on human-tied pairs, where length is the only difference
    • Self-preference delta against the human baseline
    • Sample-size and label-degeneracy checks, so a confident claim is not built on 20 rows

    Runs on the standard library. JSONL in, Markdown and JSON out, exit 1 at a severity threshold for CI.

    The shipped failing file is 80 verdicts with a judge near chance (kappa −0.07), 23% position flips, 75% self-preference and 58% longer-wins on human ties. The clean file is 60 verdicts with kappa 0.61, no flips and no length bias.

    Good to know: agreement with humans is not validity — the humans can share the judge's blind spot, and a stable judge can be consistently wrong. Kappa depends on class marginals, so a low value can reflect a skewed sample as well as a bad judge. The bands used here are triage heuristics, not a published standard. Length and model fields are optional; missing fields are reported, never imputed. This audits verdicts; it cannot tell you whether the rubric measures quality.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Kappa depends on class marginals and may be low on skewed samples. Cannot verify rubric validity. Requires Python 3.10+.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 1 day ago

    • Free with an Agensi account

    Permissions

    Terminal / Shell
    Read Files
    Write Files
    Listed1 day ago

    What's inside

    Frequently Asked Questions

    This skill analyzes your LLM judge's verdicts against human-labeled gold sets to detect hidden flaws like position bias, verbosity preference, and self-model favoritism that raw accuracy scores often miss.

    The skill requires a JSONL file containing your judge's verdicts and human labels; it operates locally and does not require any external LLM API calls or API keys to perform the analysis.

    Yes, you can integrate it into your CI/CD pipelines using the `--fail-on` flag to automatically block deployments if metrics like Cohen's Kappa or bias thresholds fall below your requirements.

    You will receive the `judge_check.py` execution script, comprehensive documentation on interpreting bias metrics, and sample JSONL templates to help you format your evaluation data.

    This skill is model-agnostic and works with verdicts from any LLM (e.g., GPT-4, Claude, Llama) as long as the output is formatted in the supported JSONL structure.

    New in LLM Ops

    Free