Works with the AI Agents you already use
AI Before/After: Compare a Prompt Change on One Sample
**Says what changes and what will be measured** (one thing that should improve, one that must not get worse) before anything runs.
Free
AI Before/After: Compare a Prompt Change on One Sample
Example session with this skill installed
문의 분류 프롬프트 바꾸기 전에 기준 떠 두고, 바꾼 뒤에 전후 비교해줘. (Take a baseline before I change the support-ticket classifier prompt, then compare after.)
- Read your context and instructions
- Compiled the ai before after
Summary
Ran the bundled example (10 labelled support messages, rule-based before/after modules, no API calls). The change made things worse: accuracy 9/10 -> 7/10, one empty output, two schema violations, and one number that is not in the input. Recommend reverting or fixing before shipping.
Plan (shown first)
Change: the classifier's rules. Should improve: fewer messages falling into other. Must not get worse: schema violations stay at 0. Labels exist, so accuracy is reported.
compare.mjs table (real run)
| 항목 | 전 | 후 | 차이 |
|---|---|---|---|
| 건수 | 10 | 10 | |
| 실패(에러) | 0 | 0 | 0 |
| 빈 출력 | 0 | 1 (10.0%) | +1 |
| 형식 위반(스키마) | 0 | 2 (20.0%) | +2 |
| 길이 평균 · 중앙 | 22 · 26 | 26 · 27 | +4 · +1 |
| 입력에 없는 숫자가 나온 건 | 0 | 1 (10.0%) | +1 |
| 정확도 | 9/10 (90.0%) | 7/10 (70.0%) | −2건 |
(건수 = rows, 빈 출력 = empty output, 형식 위반 = schema violation, 길이 = length, 입력에 없는 숫자 = number not in the input, 정확도 = accuracy.)
Label distribution: other 3 -> 1, question 2 -> 3, complaint 2 -> 1, a new label refund 0 -> 1, no label 0 -> 1.
- Labels changed: s03 request->refund, s04 other->question, s07 complaint->request, s10 other->(empty)
- Newly right: s04 · Newly wrong: s03, s07, s10
- Schema: s03 label "refund" is not an allowed value; s10 output is null
- Number not in input: s02 ("10")
One row is 10 percentage points here, so treat small moves with care.
Read by hand (--show s02,s07)
- s02: a broken cup complaint; the new output invents a "10" -> worse (made-up detail)
- s04: opening hours question now classified correctly -> better
- s07: "no reply for 3 days" moved from complaint to request -> worse
Verdict
Better: s04. Worse: s03, s07, s10, s02. Unknown: model variance (not measured; mock run).
Files: out/before.jsonl, out/after.jsonl
Connects securely to your tools. The creator never sees your data.
About this skill
"The new prompt feels better" is not a measurement. This skill makes the before/after comparison cheap enough to do every time: freeze a sample, run it through your app's own LLM function before the change, run it again after, and read one table plus a handful of changed cases.
What it does
- Says what changes and what will be measured (one thing that should improve, one that must not get worse) before anything runs. If there are no gold labels, it says up front that accuracy will not be reported.
- Builds a sample of 20-50 inputs from your data: short, long, common, ambiguous, empty, known failures. Personal data masked (
<name>,<phone>). Kept out of git. - Writes a small adapter that calls your app's function:
export async function predict(input, ctx). Real model calls happen only whenctx.realis true (REAL=1), so prompt building, parsing and post-processing still run for free with a fake model. run.mjswrites results as JSONL with mode, git commit and a sample hash. Concurrency, retries, timeouts, token usage,--dry(count and input size for a cost estimate), and it refuses to overwrite a baseline without--force.compare.mjsprints one markdown table: count, errors, empty outputs, schema violations, length mean/median, label distribution before vs after, ids whose output or label changed, numbers and English words that appear in the output but not in the input, banned patterns, latency, retries, tokens. Accuracy only on labelled rows, with newly right / newly wrong ids. It warns when the two runs used different samples or modes.- Reads 3-8 changed cases and reports: better, worse, unknown.
Rules it follows
- No gold labels -> no accuracy or recall numbers, and no "more accurate". Printing a number there would be fiction.
- Real model calls only after you approve the cost estimate. Retries cost money too.
- Does not paste the sample or full outputs into the chat.
- Runs model variance (same code twice) when you agree, so small differences are not over-read.
Zero dependencies. The example/ folder runs end to end with rule-based before/after modules and no API calls.
Instructions are in Korean; the scripts and table work for any language.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Your LLM feature must be callable as one function; you (or the agent) write a small adapter. The 'not in the input' check catches digits and English words only, not Korean proper nouns or numbers written in words. A timed-out call is not cancelled, so a paid call may still be billed. With 20-50 samples, small differences can be noise; the skill says so instead of declaring a winner. Instructions are in Korean.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
1 use
Times used to date
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 16 days ago
- Free with an Agensi account
- Software
Node.js 18+
Git