Works with the AI tools you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+22 more

    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    2

    Find position, verbosity, self-preference, format, and calibration drift before you trust an AI-generated score.

    $15

    Secure checkout via Stripe

    30-day refund guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

    Example session with this skill installed

    Audit this pairwise LLM judge. A GPT-family model scores two chatbot answers from 1–5. Option A is always shown first, answer lengths vary by about 3x, and I have 40 verdicts. Tell me whether the results are tilted and rewrite the judge prompt.

    • Read your context and instructions
    • Compiled the llm judge bias

    B3 POSITION — TILT CONFIRMED
    Evidence: option A wins 71% of 40 comparisons; a 10-pair swap test flips 4 verdicts.
    Fix: randomize order, judge each pair in both positions, and keep only stable verdicts.

    B2 VERBOSITY — EXPOSED
    Evidence: the longer answer wins 27 of 40 pairs; the rubric has no length guard.
    Fix: judge substance separately from length and add length-controlled comparisons.

    Residual risk: this calibration reduces measurement bias; it does not certify the judge as bias-free.

    Connects securely to your tools. The creator never sees your data.

    What you get

    Detect position bias in pairwise model comparisonsMitigate verbosity bias in automated content gradingCalibrate scoring rubrics against self-preference in model evaluationGenerate research-backed judge prompts with validated countermeasures

    About this skill

    What it does

    LLM-as-judge results can look precise while the evaluation setup quietly favors the first option, the longer answer, a familiar model family, or a preferred format. This skill audits an existing AI judge against five bias families and rewrites the judging apparatus with explicit mitigation rules.

    What you get

    • One evidence-linked finding card per bias family.
    • An honest magnitude statement tied to the available sample.
    • A calibrated judge prompt and operating rules.
    • A residual-risk list that stays explicit.
    • A deterministic check for sample-size, range, mitigation-tag, and people-judging controls.

    Worked path included

    No usable input triggers a labeled demonstration on a built-in pairwise A/B judge. The package also includes a complete worked example and a universal prompt edition for capable chat models that do not load skills directly.

    Known limitations

    This calibrates an existing judging setup. It does not build a full eval suite, write the content being judged, prove that a judge is bias-free, or make automated decisions about people safe by itself.

    How to install

    Works the same in every agent - Claude, Cursor, Codex, Copilot and 20+ more.

    ~30 seconds
    1. 1

      Download the ZIP

      Free skills download straight away. Paid skills unlock right after purchase.

    2. 2

      Unzip into your skills folder

      Every agent reads skills from one folder on your machine. Drop the unzipped folder in there.

    3. 3

      Ask your agent to use it

      Restart the agent if it was already running. It picks the skill up automatically - no config needed.

    Skills folder by agent

    Click the path to copy it. Create the folder if it does not exist yet.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    30-day refund

    Not a fit? Get your money back

    Trust & safety

    Security scanned

    Verified clean 1 month ago

    • Passed all security checks, Safe to install

    Listed1 month ago

    What's inside

    Frequently Asked Questions