Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+21 moreClaude CodeCursorCodex CLIMuseDots
Manus
Grok
Grok Bot+18 agentsAI Agent Testing and Evaluation Skill Bundle
A technical evaluation toolkit to compare agent runs, build regression tests, and audit claim provenance.
$9.99
AI Agent Testing and Evaluation Skill Bundle
Example session with this skill installed
I have two agent outputs for a legal summary task. Run A missed the statute of limitations. Run B included it but hallucinated the date. Compare them and create a regression test for Run B.
- Read your context and instructions
- Compiled the ai agent testing
I have analyzed the two agent runs for the legal summary task. Run A failed on completeness, while Run B failed on correctness regarding temporal data.
Agent Run Comparison: Legal Summary Task
| Metric | Run A (Completion Failure) | Run B (Accuracy Failure) |
|---|---|---|
| Instruction Adherence | Failed: Omitted statute of limitations. | Passed: Included all required sections. |
| Correctness | N/A (Missing data) | Failed: Incorrect date for statute expiry. |
| Tool-Use Quality | Nominal | Nominal |
| Failure Mode | Information Retrieval Gap | Hallucination / Fact-Extraction Error |
| Conclusion | Run B is preferred for structure but requires a provenance gate to fix date accuracy. |
Regression Test: Legal Date Extraction
test_name: test_statute_of_limitations_accuracy
scenario: Extracting filing deadlines from civil complaint documents.
input_fixture: |
"The cause of action accrued on 2021-05-10. Under local code section 402,
the plaintiff must file within three years of accrual."
expected_behavior: Agent must identify the date as 2024-05-10.
actual_behavior: Agent reported 2024-06-12 in Run B.
failure_condition: Output date does not match (Accrual Date + 3 Years).
severity: High (Legal Compliance)
verification_method: Exact date match against calculated ground truth.
Next steps
- Update the system prompt for Run B to emphasize date calculation logic.
- Add the
test_statute_of_limitations_accuracycase to your CI/CD evaluation suite. - Run the Provenance Gate on Run B to see if the hallucination stemmed from the source or the LLM logic.
Connects securely to your tools. The creator never sees your data.
About this skill
Testing AI agents often results in a "vibe-check" instead of rigorous QA. This skill replaces subjective observations with structured evaluation protocols for agent runs, tool-use quality, and safety. It provides a technical framework to measure instruction adherence and convert edge-case failures into permanent regression tests.
What it does
- Agent Run Comparison analyzes multiple outputs against the same prompt to identify trade-offs in correctness and tool execution.
- Regression Test Generation converts real-world agent failures into structured test cases with specific verification criteria.
- Dataset Construction transforms raw logs into curated evaluation records including ground-truth evidence and difficulty ratings.
- Provenance Auditing verifies every agent claim against supplied sources to detect hallucinations or missing citations.
- Content Inspection scans retrieved data for prompt injections, instruction/data confusion, and malicious directives.
How it works
- Provide the agent logs, source documents, or task outputs you need to evaluate.
- Specify the evaluation mode such as a comparison, a provenance gate, or a safety inspection.
- The skill normalizes the evidence and applies objective criteria based on the selected mode.
- Review the structured findings, including support statuses, risk categories, or new test fixtures.
Frameworks & tools
This skill is framework-agnostic and works with logs from LangChain, AutoGPT, CrewAI, or custom LLM implementations. It generates structured data suitable for unit testing suites or evaluation benches.
Why this beats prompting it yourself
Generic prompts often fail to distinguish between agent instructions and the data being processed. This skill enforces a strict separation between evidence and assumptions, ensuring that failures are documented as reproducible bugs rather than one-off errors.
Use cases
- Compare two different model versions performing the same tool-calling task.
- Generate a regression suite after a high-severity production failure.
- Audit RAG outputs to ensure every claim is traceable to the vector database.
- Inspect untrusted web-scraped content for hidden prompt injection attacks.
Known limitations
This skill requires the user to provide the original logs or evidence; it cannot observe live agent execution or fetch external data independently. Evaluation is limited to the context provided.
Requires manual provision of agent logs or source text. Cannot perform live runtime monitoring or intercept real-time API calls. Determinism depends on the clarity of supplied evidence.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before any payment.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Permissions
No special permissions declared or detected