Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    AI Before/After: Compare a Prompt Change on One Sample

    1

    **Says what changes and what will be measured** (one thing that should improve, one that must not get worse) before anything runs.

    Free

    1 useSecurity scanned
    AI Before/After: Compare a Prompt Change on One Sample

    AI Before/After: Compare a Prompt Change on One Sample

    Example session with this skill installed

    문의 분류 프롬프트 바꾸기 전에 기준 떠 두고, 바꾼 뒤에 전후 비교해줘. (Take a baseline before I change the support-ticket classifier prompt, then compare after.)

    • Read your context and instructions
    • Compiled the ai before after

    Summary

    Ran the bundled example (10 labelled support messages, rule-based before/after modules, no API calls). The change made things worse: accuracy 9/10 -> 7/10, one empty output, two schema violations, and one number that is not in the input. Recommend reverting or fixing before shipping.

    Plan (shown first)

    Change: the classifier's rules. Should improve: fewer messages falling into other. Must not get worse: schema violations stay at 0. Labels exist, so accuracy is reported.

    compare.mjs table (real run)

    항목전후차이
    건수1010
    실패(에러)000
    빈 출력01 (10.0%)+1
    형식 위반(스키마)02 (20.0%)+2
    길이 평균 · 중앙22 · 2626 · 27+4 · +1
    입력에 없는 숫자가 나온 건01 (10.0%)+1
    정확도9/10 (90.0%)7/10 (70.0%)−2건

    (건수 = rows, 빈 출력 = empty output, 형식 위반 = schema violation, 길이 = length, 입력에 없는 숫자 = number not in the input, 정확도 = accuracy.)

    Label distribution: other 3 -> 1, question 2 -> 3, complaint 2 -> 1, a new label refund 0 -> 1, no label 0 -> 1.

    • Labels changed: s03 request->refund, s04 other->question, s07 complaint->request, s10 other->(empty)
    • Newly right: s04 · Newly wrong: s03, s07, s10
    • Schema: s03 label "refund" is not an allowed value; s10 output is null
    • Number not in input: s02 ("10")

    One row is 10 percentage points here, so treat small moves with care.

    Read by hand (--show s02,s07)

    • s02: a broken cup complaint; the new output invents a "10" -> worse (made-up detail)
    • s04: opening hours question now classified correctly -> better
    • s07: "no reply for 3 days" moved from complaint to request -> worse

    Verdict

    Better: s04. Worse: s03, s07, s10, s02. Unknown: model variance (not measured; mock run).
    Files: out/before.jsonl, out/after.jsonl

    Connects securely to your tools. The creator never sees your data.

    About this skill

    "The new prompt feels better" is not a measurement. This skill makes the before/after comparison cheap enough to do every time: freeze a sample, run it through your app's own LLM function before the change, run it again after, and read one table plus a handful of changed cases.

    What it does

    1. Says what changes and what will be measured (one thing that should improve, one that must not get worse) before anything runs. If there are no gold labels, it says up front that accuracy will not be reported.
    2. Builds a sample of 20-50 inputs from your data: short, long, common, ambiguous, empty, known failures. Personal data masked (<name>, <phone>). Kept out of git.
    3. Writes a small adapter that calls your app's function: export async function predict(input, ctx). Real model calls happen only when ctx.real is true (REAL=1), so prompt building, parsing and post-processing still run for free with a fake model.
    4. run.mjs writes results as JSONL with mode, git commit and a sample hash. Concurrency, retries, timeouts, token usage, --dry (count and input size for a cost estimate), and it refuses to overwrite a baseline without --force.
    5. compare.mjs prints one markdown table: count, errors, empty outputs, schema violations, length mean/median, label distribution before vs after, ids whose output or label changed, numbers and English words that appear in the output but not in the input, banned patterns, latency, retries, tokens. Accuracy only on labelled rows, with newly right / newly wrong ids. It warns when the two runs used different samples or modes.
    6. Reads 3-8 changed cases and reports: better, worse, unknown.

    Rules it follows

    • No gold labels -> no accuracy or recall numbers, and no "more accurate". Printing a number there would be fiction.
    • Real model calls only after you approve the cost estimate. Retries cost money too.
    • Does not paste the sample or full outputs into the chat.
    • Runs model variance (same code twice) when you agree, so small differences are not over-read.

    Zero dependencies. The example/ folder runs end to end with rule-based before/after modules and no API calls.

    Instructions are in Korean; the scripts and table work for any language.

    Source: https://github.com/Stophyuk/solutionzip-skills

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Your LLM feature must be callable as one function; you (or the agent) write a small adapter. The 'not in the input' check catches digits and English words only, not Korean proper nouns or numbers written in words. A timed-out call is not cancelled, so a paid call may still be billed. With 20-50 samples, small differences can be noise; the skill says so instead of declaring a winner. Instructions are in Korean.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    1 use

    Times used to date

    Free to use

    No account required to browse

    Trust & safety

    Security scan

    Security scan passed 16 days ago

    • Free with an Agensi account

    Software
    Node.js 18+Git

    Permissions

    Read Files
    Write Files
    Terminal / Shell
    Environment Variables
    Listed16 days ago

    What's inside

    Frequently Asked Questions

    Without known right answers, an accuracy number would be made up. It reports what can be measured without labels: empty and malformed outputs, length, how the label distribution moved, which ids changed, and numbers that appear in the output but not in the input.

    Not by default. Runs use a mock unless REAL=1 is set, and the skill gives a cost estimate (--dry or a one-row run) and waits for your yes before any real run.

    Yes. It calls your own function, so whatever SDK or model your app uses is what gets measured. The scripts never call a provider directly.

    While it doesn't require a specific platform, it is designed for developers who can integrate a simple JS shim to bridge their application logic with the skill's comparison engine.

    Mock testing allows you to validate your prompt assembly, parsing, and post-processing logic without making actual API calls, saving you significant time and credit costs during the development phase.

    For statistically significant results, we recommend a sample size of 20 to 50 items; smaller datasets may not provide enough data to identify clear performance trends or regressions.

    New in LLM Ops

    Free