- Home
- Skills
- Agents & Orchestration
- LLM Judge Bias Calibrator — Audit AI Evaluation Fairness
Works with the AI Agents you already use
LLM Judge Bias Calibrator — Audit AI Evaluation Fairness
Find position, verbosity, self-preference, format, and calibration drift before you trust an AI-generated score.
$15
LLM Judge Bias Calibrator — Audit AI Evaluation Fairness
Example session with this skill installed
Audit this pairwise LLM judge. A GPT-family model scores two chatbot answers from 1–5. Option A is always shown first, answer lengths vary by about 3x, and I have 40 verdicts. Tell me whether the results are tilted and rewrite the judge prompt.
- Read your context and instructions
- Compiled the llm judge bias
B3 POSITION — TILT CONFIRMED
Evidence: option A wins 71% of 40 comparisons; a 10-pair swap test flips 4 verdicts.
Fix: randomize order, judge each pair in both positions, and keep only stable verdicts.
B2 VERBOSITY — EXPOSED
Evidence: the longer answer wins 27 of 40 pairs; the rubric has no length guard.
Fix: judge substance separately from length and add length-controlled comparisons.
Residual risk: this calibration reduces measurement bias; it does not certify the judge as bias-free.
Connects securely to your tools. The creator never sees your data.
About this skill
What it does
LLM-as-judge results can look precise while the evaluation setup quietly favors the first option, the longer answer, a familiar model family, or a preferred format. This skill audits an existing AI judge against five bias families and rewrites the judging apparatus with explicit mitigation rules.
What you get
- One evidence-linked finding card per bias family.
- An honest magnitude statement tied to the available sample.
- A calibrated judge prompt and operating rules.
- A residual-risk list that stays explicit.
- A deterministic check for sample-size, range, mitigation-tag, and people-judging controls.
Worked path included
No usable input triggers a labeled demonstration on a built-in pairwise A/B judge. The package also includes a complete worked example and a universal prompt edition for capable chat models that do not load skills directly.
Known limitations
This calibrates an existing judging setup. It does not build a full eval suite, write the content being judged, prove that a judge is bias-free, or make automated decisions about people safe by itself.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.
Requires an existing judge prompt or rubric; it does not design an eval program from zero. Fewer than five scored items produces design-only exposure analysis, not measured tilt. Results below the numeric sample floor are reported directionally rather than as percentages. Human-impact uses such as CV screening or admissions require independent fairness, legal, and human-review controls.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 3 months ago
Permissions
No special permissions declared or detected