Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    2

    Find position, verbosity, self-preference, format, and calibration drift before you trust an AI-generated score.

    $15

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    Example session with this skill installed

    Audit this pairwise LLM judge. A GPT-family model scores two chatbot answers from 1–5. Option A is always shown first, answer lengths vary by about 3x, and I have 40 verdicts. Tell me whether the results are tilted and rewrite the judge prompt.

    • Read your context and instructions
    • Compiled the llm judge bias

    B3 POSITION — TILT CONFIRMED
    Evidence: option A wins 71% of 40 comparisons; a 10-pair swap test flips 4 verdicts.
    Fix: randomize order, judge each pair in both positions, and keep only stable verdicts.

    B2 VERBOSITY — EXPOSED
    Evidence: the longer answer wins 27 of 40 pairs; the rubric has no length guard.
    Fix: judge substance separately from length and add length-controlled comparisons.

    Residual risk: this calibration reduces measurement bias; it does not certify the judge as bias-free.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    What it does

    LLM-as-judge results can look precise while the evaluation setup quietly favors the first option, the longer answer, a familiar model family, or a preferred format. This skill audits an existing AI judge against five bias families and rewrites the judging apparatus with explicit mitigation rules.

    What you get

    • One evidence-linked finding card per bias family.
    • An honest magnitude statement tied to the available sample.
    • A calibrated judge prompt and operating rules.
    • A residual-risk list that stays explicit.
    • A deterministic check for sample-size, range, mitigation-tag, and people-judging controls.

    Worked path included

    No usable input triggers a labeled demonstration on a built-in pairwise A/B judge. The package also includes a complete worked example and a universal prompt edition for capable chat models that do not load skills directly.

    Known limitations

    This calibrates an existing judging setup. It does not build a full eval suite, write the content being judged, prove that a judge is bias-free, or make automated decisions about people safe by itself.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.

    Requires an existing judge prompt or rubric; it does not design an eval program from zero. Fewer than five scored items produces design-only exposure analysis, not measured tilt. Results below the numeric sample floor are reported directionally rather than as percentages. Human-impact uses such as CV screening or admissions require independent fairness, legal, and human-review controls.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 3 months ago

    Permissions

    No special permissions declared or detected

    Listed3 months ago

    What's inside

    Frequently Asked Questions

    Provide the judge prompt or rubric, what is being scored, the judge model, and a sample of scored results. Without a usable sample, the skill runs in design-only mode.

    No. Fewer than five items stays design-only, and samples below the numeric floor are reported directionally rather than as percentages.

    No. Every audit includes residual risks. The skill reduces documented measurement biases but never certifies a judge as bias-free.

    No. It calibrates an existing judging setup. Designing a full eval suite, generating test data, and running production evaluations are outside its scope.

    It can flag measurement risks, but people-judging requires independent fairness, legal, disclosure, and human-review controls. This calibration does not make automated decisions about people safe.

    New in Agents & Orchestration