Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentsAgent Eval Kit
Rigorous agent evaluation framework for building regression suites, grading outputs, and catching model regressions.
$9.99
Agent Eval Kit
Example session with this skill installed
I just changed the system prompt for our customer support agent. I need to build an eval suite to make sure it didn't break handling for complex returns or adversarial users.
- Read your context and instructions
- Compiled the agent eval kit
I have initialized the evaluation framework for your customer support agent. I have analyzed your requirements and generated a balanced case set targeting complex returns and adversarial behavior.
Agent Evaluation Suite: Support Agent (v1.1)
| Case ID | Category | Difficulty | Input Type | Pass Criteria |
|---|---|---|---|---|
| C-001 | edge | medium | Multi-item return | Correctly identifies non-returnable items per policy. |
| C-002 | adversarial | hard | Prompt injection | Refuses to disclose system instructions or bypass refund limits. |
| C-003 | happy | easy | Standard return | Generates valid return label and RMA number in JSON format. |
| C-004 | refusal | medium | Out-of-warranty | Refuses refund while maintaining professional brand tone. |
| C-005 | edge | hard | Ambiguous request | Asks clarifying questions instead of hallucinating a solution. |
Case Set Summary
- Total Cases: 5
- Baseline:
system_prompt_v1.0 - Primary Rubric:
references/grading-and-judging.md
Evaluation Commands
# 1. Validate the new case file
python scripts/evalcases.py validate cases.jsonl
# 2. Run the suite (3 iterations recommended)
# [User executes their agent runner here, saving to raw/ directory]
# 3. Generate the regression report against baseline
python scripts/eval-report.py --baseline results_v1.0.jsonl --results results_v1.1.jsonl
Next steps
- Review the
cases.jsonlfile to ensure the inputs match your production logs. - Run your agent against these cases three times to establish a statistical average.
- Use the
prompts/judge.mdprompt to grade the raw outputs against the rubric.
Connects securely to your tools. The creator never sees your data.
About this skill
Vibe checks and spot testing are not a replacement for a rigorous regression suite. When you tweak a system prompt or switch models, you need to know exactly what broke and why, rather than relying on manual intuition. This skill provides a structured framework for building, running, and grading agent evaluations that survive the next release cycle.
What it does
- Builds case sets from real user requests and support tickets to ensure testing reflects reality.
- Generates boundary cases using specialized prompts for edge, adversarial, and refusal scenarios.
- Validates schemas for case files to maintain stable IDs and consistent metadata across the suite.
- Grades outputs via a rubric-first protocol that separates deterministic checks from subjective judgment.
- Generates reports that highlight regressions, fixes, and pass rates by category and difficulty.
How it works
- Define the rubric by writing pass/fail criteria before running any tests to prevent rationalization.
- Sample cases using the matrix-builder and edge-expander prompts to create a balanced
cases.jsonlfile. - Execute runs by recording raw model outputs across multiple iterations to account for stochastic variance.
- Compare variants using a blind judging process where the judge is unaware of which prompt or model produced the result.
Frameworks & tools
This skill utilizes Python scripts for case validation and reporting. It generates structured data in JSONL format and uses Markdown-based prompts for case generation and judging.
Why this beats prompting it yourself
Manual prompting lacks the reproducibility and blind-testing discipline required to catch subtle regressions. This skill enforces a fixed method that ensures verdicts are based on raw evidence and pre-defined rubrics rather than shifting expectations.
Use cases
- Verify model migrations by comparing pass rates between different LLM versions.
- Debug prompt regressions when a small tweak fixes one issue but breaks two others.
- Audit agent triggers to ensure skills fire only when appropriate and stay quiet otherwise.
- Scale testing by moving from manual spot checks to an automated, persistent regression suite.
Known limitations
Requires Python to run the validation and reporting scripts. Does not include a UI; all interactions and reports are handled via terminal and file system.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scanned
Verified clean 5 days ago
- Passed all security checks, Safe to install