More screenshots
Works with the AI Agents you already use
LLM Eval Run Comparability & Reproducibility Receipt
Compare two local LLM evaluation manifests and issue a deterministic receipt showing whether their reported scores are structurally comparable.
$9
LLM Eval Run Comparability & Reproducibility Receipt
Example session with this skill installed
Compare baseline.json with candidate.json under policy.json and write a new eval-comparability-packet. Tell me whether the score delta is structurally comparable, and list all controlled and uncontrolled changes.
- Read your context and instructions
- Compiled the llm eval run
- Generated the data export
LLM Eval Comparability Receipt
Status: BLOCK
Run ID: 1eae0fcd9fc17dacc3d0
The two supplied runs are not yet structurally comparable. One uncontrolled provenance field changed, so the reported score delta should not be interpreted until the BLOCK item is resolved.
| Check | Baseline | Candidate | Controlled | Decision |
|---|---|---|---|---|
| dataset_sha256 | aaaa…aaaa | ffff…ffff | No | REVIEW |
| prompt_sha256 | bbbb…bbbb | eeee…eeee | Yes | REVIEW |
| repetitions | 5 | 5 | No change | READY |
| temperature | 0 | 0 | No change | READY |
Summary
- Fields compared: 18
- Controlled changes: 1
- Uncontrolled changes: 1
- Baseline repetitions: 5
- Candidate repetitions: 5
- Reported score delta: 0.05
Issues
- BLOCK — UNCONTROLLED_CHANGE_LIMIT_EXCEEDED: the policy does not allow this uncontrolled dataset change.
- REVIEW — CONTROLLED_FIELD_CHANGED: the prompt hash changed and was declared controlled.
- REVIEW — UNCONTROLLED_FIELD_CHANGED: the dataset hash changed without control.
Packet written
result.jsoncomparability-matrix.csvprovenance-delta.csvissues.jsonlEVAL_COMPARABILITY.mdevidence-receipt.json
Resolve every BLOCK item and document each controlled change before interpreting the score difference. This receipt compares supplied provenance; it does not rerun an evaluation, establish statistical significance or decide which system is better.
comparability-matrix.csv
CSV · data export
Example file from a real run - the skill writes it into your workspace.
Connects securely to your tools. The creator never sees your data.
About this skill
The problem
Two LLM or RAG evaluation runs can show different scores even when their datasets, prompts, model settings or evidence were not held constant. Without a structural comparison, teams may treat an uncontrolled change as a real improvement.
What it does
This downloadable local package compares two buyer-supplied evaluation manifests under a policy and writes a deterministic review packet. It includes an installable Python CLI, a local stdio MCP adapter and a companion Agent Skill.
- Compares 18 provenance and configuration fields.
- Separates controlled changes from uncontrolled changes.
- Produces result.json, comparability-matrix.csv, provenance-delta.csv, issues.jsonl, EVAL_COMPARABILITY.md and evidence-receipt.json.
- Rejects malformed inputs and unsafe paths, never overwrites an existing packet and binds inputs and outputs with SHA-256.
Best for
- AI product teams and QA consultants reviewing LLM or RAG experiment runs.
- Agencies preparing a reproducible client handoff.
- Teams that need evidence before accepting a reported score change.
Privacy and requirements
Processing stays local. No API key, login, upload, telemetry or live external action is included. Requires Windows 11 and Python 3.11–3.14. The local MCP adapter optionally uses mcp==2.2.0.
Known limitations
The tool does not execute models, design benchmarks, recompute metrics, claim statistical significance, judge result quality, log into vendors or reproduce an experiment. It evaluates structural comparability only.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Compares supplied structured manifests only. It does not execute models, design benchmarks, recompute metrics, establish statistical significance, judge which system is better, log into vendors or reproduce an experiment. Input files must follow the documented schema.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 2 months ago
- Software
Python
Permissions
File Scopes
Reads only the selected local baseline, candidate and policy JSON files, then writes a new evaluation-comparability packet. It does not use the browser, network, credentials or environment variables.