More screenshots
Works with the AI Agents you already use
llm eval assertion grader
Grade LLM outputs against per-case assertions (contains, regex, JSON keys, word limits, no-refusal), get a weighted pass rate, per-tag results and a regression diff vs your last run.
$8.90
llm eval assertion grader
Example session with this skill installed
Grade the outputs in v2-results.jsonl against cases.jsonl, using v1-baseline.json as the reference. Set a minimum pass rate of 90%.
- Read your context and instructions
- Compiled the llm eval assertion
I have processed the evaluation of v2-results.jsonl. The run failed the 90% quality gate with a weighted score of 84.5%. A regression was detected in the json-formatting tag.
LLM Evaluation Report
| Metric | Value | Status |
|---|---|---|
| Weighted Pass Rate | 84.5% | ❌ FAIL (Gate: 90%) |
| Total Cases | 100 | - |
| Passed / Failed | 82 / 18 | - |
| Regression vs Baseline | -5.2% | ⚠️ DECLINE |
Results by Tag
| Tag | Pass Rate | Change |
|---|---|---|
refusal-check | 100% | 0% |
json-formatting | 70% | -15% |
conciseness | 92% | +2% |
Top Failure Assertions
- json_keys: 8 cases failed. The model omitted the
required_tokenskey in the response. - max_words: 6 cases failed. Output exceeded 150-word limit by an average of 12 words.
- regex: 4 cases failed. Failed to match standard UUID format in
transaction_id.
Next steps
- Inspect the 8
json_keysfailures to adjust the system prompt for stricter schema adherence. - Increase the
max_wordslimit incases.jsonlif the longer outputs are actually preferred. - Re-run the evaluation using
python3 scripts/evalgrade.py --cases cases.jsonl --outputs v3-results.jsonl --baseline v2-results.json.
Connects securely to your tools. The creator never sees your data.
About this skill
Know if your prompt change made things better or worse in one command - no eval framework needed.
Write test cases as JSONL with simple assertions, run your model however you like, and grade the outputs offline. The grader reports a weighted pass rate, results by tag, every failed assertion with the reason, and which cases regressed since a saved baseline. A --min-pass gate makes it CI friendly.
What it does
- 14 assertion types: contains / not_contains / contains_all / contains_any, regex, equals, one_of, min/max words, max chars, is_json, json_keys, starts_with, no_refusal
- Weighted cases and per-tag pass rates
- Regression diff against a previous results.json
- Markdown + JSON reports, exit 1 below
--min-pass
Who it's for
Prompt engineers and AI app builders iterating on prompts, models or system messages.
Limits
Deterministic assertions only - no semantic similarity or LLM-judge scoring. Not legal/financial advice.
What's included
SKILL.md: workflow, commands, how to explain findingsscripts/: the tool (stdlib Python)references/sources.md: specs and guidance citedexamples/: sample inputs used in the demotests/: unit tests (Python 3.8 / 3.9 / 3.13)
Standard-library Python only. No API keys, no network calls, no installs.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Offline / local files only - does not fetch live URLs or call APIs.
Deterministic assertions only - no semantic similarity or LLM-judge scoring.
Regex and keyword checks can be gamed; keep cases realistic.
Not legal/financial advice.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 16 days ago
Permissions
File Scopes
Fully offline. No subprocess, no network calls, no credential access.