Works with the AI tools you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIMuseMuseDotsDots+22 more

    LLM Eval Harness

    1

    Evaluate LLM prompts and models with repeatable test suites, semantic checks, and statistical comparisons.

    $9.99

    Secure checkout via Stripe

    30-day refund guarantee

    Converts to your local currency at checkout

    Security scanned
    LLM Eval Harness

    LLM Eval Harness

    Example session with this skill installed

    Create a JSONL eval case for a customer support agent that must return a JSON object with a 'ticket_id' and cannot contain the word 'apologize'. Then show me how to run it.

    • Read your context and instructions
    • Compiled the llm eval harness

    I have drafted the evaluation case and the corresponding execution command. This setup ensures your agent follows the required JSON schema while adhering to your negative constraints.

    Evaluation Case Definition

    {"id": "support-001", "input": "Cancel my order #12345", "expected": "{\"ticket_id\": 12345}", "checks": ["json", "has_key:ticket_id", "not_contains:apologize", "max_words:50"], "tags": ["cancellation", "json-output"]}

    Execution Commands

    To run this evaluation against your local model script and generate a report:

    
# 1. Run the evaluation
python3 scripts/evals.py run cases.jsonl --cmd "python3 agent.py" -o results_v1.jsonl

# 2. Score the results and generate a Markdown report
python3 scripts/evals.py score cases.jsonl results_v1.jsonl --md report.md --min-pass 1.0

# 3. View the summary
cat report.md

    Next steps

    1. Add more variations of the "cancel" intent to the cases.jsonl file to increase test coverage.
    2. Integrate the score command into your GitHub Actions workflow to gate PRs on 100% pass rates.
    3. Use the compare command if you decide to swap the underlying model or adjust the system prompt.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Prompt tweaks and model swaps often break LLM features in ways unit tests miss. This skill provides a repeatable evaluation harness to quantify performance changes across datasets.

    What it does

    • Multi-check validation applies exact match, regex, JSON schema, word counts, and semantic similarity scores like ROUGE-L.
    • Model-agnostic execution runs tests via shell commands or HTTP endpoints to support any local or cloud provider.
    • Regression reporting generates Markdown and JSON reports with pass rates aggregated by custom tags.
    • Comparative analysis compares two different runs or prompts side-by-side using McNemar significance testing.

    How it works

    1. Define cases in a JSONL file specifying inputs, expected outputs, and specific check types like contains, json, or refusal.
    2. Execute runs by pointing the harness at your model interface using the run command.
    3. Score results against your dataset to calculate pass rates and latency metrics.
    4. Compare outputs to identify specific wins, losses, and statistical significance between model versions.

    Frameworks & tools

    Built for Python environments. Integrates with any model interface via CLI or HTTP. Uses JSONL for datasets and outputs Markdown for CI/CD reporting.

    Why this beats prompting it yourself

    Manual inspection of LLM outputs doesn't scale and misses subtle regressions in JSON formatting or word counts. This skill automates the scoring logic and statistical comparisons, providing a binary pass/fail signal for CI gates that simple prompting cannot replicate reliably.

    Use cases

    • Prompt engineering to verify if a new system prompt improves accuracy without breaking formatting.
    • Model migration to test if switching from GPT-4 to a local Llama-3 model maintains quality standards.
    • CI/CD integration to block deployments if the pass rate on a golden dataset drops below a defined threshold.

    Known limitations

    Requires users to provide their own execution shim for models. The harness does not include built-in LLM-as-a-judge capabilities.

    How to use

    Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill passes our trust checks below.

    Security scanned

    Passed our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    30-day refund

    Not a fit? Get your money back

    Trust & safety

    Security scanned

    Verified clean today

    • Passed all security checks, Safe to install

    Listedtoday

    What's inside

    Frequently Asked Questions