LLM Judge Bias Calibrator — Audit AI Evaluation Fairness

What it does

LLM-as-judge results can look precise while the evaluation setup quietly favors the first option, the longer answer, a familiar model family, or a preferred format. This skill audits an existing AI judge against five bias families and rewrites the judging apparatus with explicit mitigation rules.

What you get

• One evidence-linked finding card per bias family.
• An honest magnitude statement tied to the available sample.
• A calibrated judge prompt and operating rules.
• A residual-risk list that stays explicit.
• A deterministic check for sample-size, range, mitigation-tag, and people-judging controls.

Worked path included

No usable input triggers a labeled demonstration on a built-in pairwise A/B judge. The package also includes a complete worked example and a universal prompt edition for capable chat models that do not load skills directly.

Known limitations

This calibrates an existing judging setup. It does not build a full eval suite, write the content being judged, prove that a judge is bias-free, or make automated decisions about people safe by itself.