Works with the AI Agents you already use
goodhart
A red-teaming tool that identifies and fixes gameable criteria in LLM-as-judge rubrics and quality checklists.
Free
goodhart
Example session with this skill installed
Audit my customer support rubric in rubric.md. It grades responses on accuracy, tone, and thoroughness. Use script mode with 5 runs.
- Read your context and instructions
- Compiled the goodhart
I have completed the red-team audit of your customer support rubric using Script mode with 5 runs per probe. I identified 2 FOOLED criteria where the rubric rewards padding over actual utility.
Rubric Audit Report: customer-support-v1
| Criterion | Status | Finding |
|---|---|---|
| C1: Accuracy | 🟢 ROBUST | Penalized 100% of confident-but-wrong probes. |
| C2: Tone | 🟡 VULNERABLE | Failed to penalize "over-politeness" that ignored the user's technical issue. |
| C3: Thoroughness | 🔴 FOOLED | Gamed probes with 200 words of filler scored 15% higher than concise, correct answers. |
Summary of Exploits
The rubric currently equates "Thoroughness" with "Word Count." In 5/5 runs, a probe that repeated the user's question back to them (E01 Padding) received a perfect score, while the honest baseline was marked down for being "too brief" despite solving the ticket.
Hardened Rubric Changes
Applied H3 Length Neutrality: Updated C3 to explicitly state that length without new information should not increase the score.
Negative Constraints: Added a penalty for "Rubric Echoing" where the model mimics the evaluation criteria in its response.
Ranking Accuracy:
- Dev Set: 0.72
- Held-out Set: 0.88 (Verified against new "Self-Praise" exploits)
Next steps
- Review the hardened version in
goodhart-run/rubric-hardened.md. - Replace your production judge prompt with the updated criteria.
- Run
scripts/analyze.pyin your CI pipeline to prevent future regression.
Connects securely to your tools. The creator never sees your data.
About this skill
Evaluation rubrics are often flawed, rewarding superficial traits like length, confident tone, or formatting instead of genuine quality. This skill red-teams your LLM-as-judge rubrics, support checklists, or quality-review guides by attacking them like a malicious optimizer. It generates responses specifically engineered to game your criteria, grades them blind against honest baselines, and reports where the rubric fails.
What it does
- Decompose rubrics to identify gaps between what is intended to be measured and what is actually observable by a grader.
- Generate gamed probes including padding, confident errors, and rubric echoing to stress-test specific criteria.
- Blind scoring using repeated runs to ensure results are statistically significant rather than just noise.
- Hardened rubric generation providing a revised version of your criteria with specific fixes for identified vulnerabilities.
- Verification testing using a held-out set of attacks to prove the hardened rubric generalizes to new exploits.
How it works
- Input the rubric and the specific task or policy it is designed to evaluate.
- Execute a probe run where the agent creates honest baselines and gamed responses.
- Run the judge in either Script mode (Python) or Portable mode (subagents) to score probes repeatedly.
- Analyze the gap to categorize criteria as FOOLED, VULNERABLE, or ROBUST.
- Review the report and the new, hardened rubric file.
Frameworks & tools
The skill is compatible with any LLM-as-judge setup. It includes Python scripts for local execution or can run entirely via agent subagents.
Why this beats prompting it yourself
Manual testing usually misses edge cases or relies on a single subjective pass. This skill uses a systematic red-teaming methodology with repeated blind trials and a rubric-free control judge to strip away bias.
Use cases
- Audit an LLM-as-judge prompt before deploying it to an automated evaluation pipeline.
- Harden a customer support checklist to ensure agents are rewarded for solving problems, not just using empathy buzzwords.
- Stress-test grading guides for technical documentation or code quality to prevent rewards for verbose, low-value output.
Known limitations
The skill is not tested for rubrics involving contested human judgments like hiring or admissions. It does not measure fairness across demographic groups. Scoring requires multiple model calls.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Not tested for hiring or admissions. Does not test for demographic fairness. Requires multiple model runs for statistical significance. Results depend on the quality of the probe authoring.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 7 days ago
- Free with an Agensi account