More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    llm eval assertion grader

    1

    Grade LLM outputs against per-case assertions (contains, regex, JSON keys, word limits, no-refusal), get a weighted pass rate, per-tag results and a regression diff vs your last run.

    $8.90

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    llm eval assertion grader

    llm eval assertion grader

    Example session with this skill installed

    Grade the outputs in v2-results.jsonl against cases.jsonl, using v1-baseline.json as the reference. Set a minimum pass rate of 90%.

    • Read your context and instructions
    • Compiled the llm eval assertion

    I have processed the evaluation of v2-results.jsonl. The run failed the 90% quality gate with a weighted score of 84.5%. A regression was detected in the json-formatting tag.

    LLM Evaluation Report

    MetricValueStatus
    Weighted Pass Rate84.5%❌ FAIL (Gate: 90%)
    Total Cases100-
    Passed / Failed82 / 18-
    Regression vs Baseline-5.2%⚠️ DECLINE

    Results by Tag

    TagPass RateChange
    refusal-check100%0%
    json-formatting70%-15%
    conciseness92%+2%

    Top Failure Assertions

    1. json_keys: 8 cases failed. The model omitted the required_tokens key in the response.
    2. max_words: 6 cases failed. Output exceeded 150-word limit by an average of 12 words.
    3. regex: 4 cases failed. Failed to match standard UUID format in transaction_id.

    Next steps

    • Inspect the 8 json_keys failures to adjust the system prompt for stricter schema adherence.
    • Increase the max_words limit in cases.jsonl if the longer outputs are actually preferred.
    • Re-run the evaluation using python3 scripts/evalgrade.py --cases cases.jsonl --outputs v3-results.jsonl --baseline v2-results.json.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Know if your prompt change made things better or worse in one command - no eval framework needed.

    Write test cases as JSONL with simple assertions, run your model however you like, and grade the outputs offline. The grader reports a weighted pass rate, results by tag, every failed assertion with the reason, and which cases regressed since a saved baseline. A --min-pass gate makes it CI friendly.

    What it does

    • 14 assertion types: contains / not_contains / contains_all / contains_any, regex, equals, one_of, min/max words, max chars, is_json, json_keys, starts_with, no_refusal
    • Weighted cases and per-tag pass rates
    • Regression diff against a previous results.json
    • Markdown + JSON reports, exit 1 below --min-pass

    Who it's for

    Prompt engineers and AI app builders iterating on prompts, models or system messages.

    Limits

    Deterministic assertions only - no semantic similarity or LLM-judge scoring. Not legal/financial advice.

    What's included

    • SKILL.md: workflow, commands, how to explain findings
    • scripts/: the tool (stdlib Python)
    • references/sources.md: specs and guidance cited
    • examples/: sample inputs used in the demo
    • tests/: unit tests (Python 3.8 / 3.9 / 3.13)

    Standard-library Python only. No API keys, no network calls, no installs.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Offline / local files only - does not fetch live URLs or call APIs.
    Deterministic assertions only - no semantic similarity or LLM-judge scoring.
    Regex and keyword checks can be gamed; keep cases realistic.
    Not legal/financial advice.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 16 days ago

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    File Scopes

    llm-eval-assertion-grader/**

    Fully offline. No subprocess, no network calls, no credential access.

    Listed16 days ago

    What's inside

    Frequently Asked Questions

    No. You produce the outputs JSONL with your own model or tool; the grader is deterministic and offline.

    Only through assertions you write. There is no LLM-as-judge; that keeps results reproducible.

    JSONL with `id` and `output` fields, matched to case ids.

    New in LLM Ops