More screenshots
Works with the AI Agents you already use
Eval & Fine-Tuning Dataset Builder: Logs to Leak-Free Data
Turn real chat logs and prompts into a clean eval set or fine-tuning dataset: dedupe, redact personal data, fill coverage gaps, split without leakage, freeze the test set, export validated files with a datacard.
$9
Eval & Fine-Tuning Dataset Builder: Logs to Leak-Free Data
Example session with this skill installed
Build an eval set for our support bot from chat_logs.jsonl (OpenAI format) and faq_prompts.csv. Remove duplicates and any customer personal data, make sure it covers refusals and vague messages, and give me train/dev/test files I can trust. The bot is also given the FAQ file, so check the test set against it.
- Read your context and instructions
- Compiled the eval & fine-tuning
Built a 54-item dataset from your 37 conversations and 4 FAQ rows: 40 real items plus 14 checked cases for the gaps. The test set (11 items) is frozen, and nothing from it appears in train/dev. One test question is copied word for word from the FAQ the bot is given; details below.
Cleaning
| Step | Result |
|---|---|
| Imported | 43 items (one per user turn; 2 multi-turn) |
| Removed | 2 exact duplicates (one was also in the FAQ), 1 too short ("ok") |
| Personal data redacted | 1 email, 1 card number, 1 phone number |
| Grouped (same split always) | 5 groups: paraphrases like "How do I export my notes to PDF?" / "How can I export all of my notes into a PDF file?", plus 4 pairs I joined by hand (cancel, share, restore, sync) |
Coverage: what was missing
Real logs were 19 how-to questions and almost no hard cases. After tagging (intent x expected behavior):
| Cell | Real logs | After adding cases |
|---|---|---|
| Unsafe + Decline (prompt injection, account takeover) | 1 | 4 |
| Off-topic + Decline | 1 | 3 |
| Unclear + Clarify ("it doesnt work", "help") | 1 | 3 |
| Account & data + Answer | 2 | 3 |
All 14 added cases passed the checks (valid tags, no personal data, no near-copies). 8 cells are still below 3 items, mostly Clarify and Escalate for billing, account and sync; they're listed in coverage.md for you to decide on.
Split and leakage
split 54 kept items: train 39, dev 4, test 11
test set frozen (11 items)
similarity groups crossing splits: 0
leakage check (test vs train/dev): RESULT: CLEAN
leakage check --against faq_prompts.csv:
1.00 test r000041 ~ faq_prompts.csv row 3 ("Can two people edit the same note at once?")
Because the bot is given the FAQ, that test question measures recall of the FAQ, not reasoning. Keep it if you want to test FAQ grounding, or exclude it and re-split.
Files
dataset/train.jsonl,dev.jsonl,test.jsonl(OpenAI chat format):validatereports VALID, 39 train rows, longest ~91 tokensdataset/eval/test.jsonlwithexpected,tagsandexpected_byper itemDATACARD.md(sources, removals, redactions, composition, splits, leakage, known limits) andtest_manifest.json(hashes of the frozen test set)
Next steps
- Spot-check the 15 answers marked
expected_by: agent, starting with the ones in test. - Decide on the FAQ question above.
- Tell me which of the 8 remaining thin cells matter and I'll add cases for them.
Connects securely to your tools. The creator never sees your data.
About this skill
Most LLM eval sets are a spreadsheet of questions someone typed in one afternoon: full of near-duplicates, missing the cases that actually break things, with paraphrases of the test questions sitting in the training data and a customer's email address in row 212. This skill makes your agent build the dataset the careful way: start from real traffic, clean it, show you which kinds of requests are missing, fill those gaps, split it so nothing leaks, freeze the test set, and hand you validated files with a datacard.
No API key and no extra cost: the agent you already use reads and writes the content. A bundled Python script (standard library only) does everything that has to be exact.
What it does
- Imports the formats you actually have: OpenAI chat JSONL, ShareGPT, the ChatGPT data export, Alpaca, CSV/TSV/Excel and plain text. Conversations become one item per user turn, with the history and system prompt kept.
- Cleans: removes exact and near-duplicates, drops empty or oversized items, and redacts emails, phone numbers, card numbers, IPs, IBANs and API keys.
- Prevents leakage at the source: similar items, paraphrases you point out, and turns of the same conversation are grouped so they never land on both sides of the split.
- Shows what's missing: every item is tagged (for example what the user wants x what the assistant should do) in validated batches, then a coverage matrix lists the thin cells, such as too few refusals, prompt-injection attempts or vague messages.
- Fills the gaps with checked cases: the agent writes realistic cases for the cells that matter; a file is rejected if any case has an invalid tag, personal data, or near-copies an existing item.
- Writes missing reference answers in checked batches, marked
expected_by: agentso you know what to spot-check. - Splits and freezes: stratified train/dev/test that keeps groups together; the test set is locked with content hashes and later edits are refused.
- Checks leakage: test vs. train/dev, test vs. text the model sees (system prompt, few-shot examples, docs), and shared answers.
- Exports: OpenAI fine-tuning JSONL, eval JSONL or CSV, plus
DATACARD.mdandtest_manifest.json, and a validator for the files.
How it works
- Import logs and exports into one item list.
- Clean, then read samples and the similarity groups; join paraphrases the automatic grouping missed.
- Tag every item and look at the coverage matrix; mark impossible combinations so only real gaps remain.
- Fill gaps with realistic, checked cases; write missing reference answers.
- Split (group-aware, stratified) and freeze the test set.
- Check leakage against train/dev and your prompts or few-shot examples.
- Export and validate; the datacard records sources, removals, redactions, composition, splits, leakage and known limits.
Use cases
- Building a golden test set for a chatbot, RAG system or agent from production logs.
- Preparing supervised fine-tuning (SFT) data in OpenAI chat format.
- Auditing an existing eval set for duplicates, leakage and missing safety cases.
- Keeping a frozen, hash-verified test set so scores stay comparable between model versions.
Known limitations
- Grouping and leakage checks are word-based: they catch copies, light edits and rewordings that keep the key terms. A paraphrase with entirely different words is caught only when the agent joins it by hand while reading.
- Personal-data redaction is pattern-based; names and street addresses are not detected.
- Agent-written answers and cases are marked as such and should be spot-checked, especially in the test set.
- Cleaning takes about 45 seconds per 10,000 rows; tagging runs at the speed your agent reads.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Paraphrase and leakage detection is word-based: rewordings with entirely different words are caught only when the agent joins them by hand. Personal-data redaction is pattern-based and does not detect names or street addresses. Agent-written answers and added cases are marked and should be spot-checked. Labels come from fixed tag sets; it does not score your model, it builds the data you score it on.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 5 days ago
- Software
Python 3.8+
- Optional
- openpyxl (only for Excel input)
No API key needed: your own agent does the tagging and writing. Bring your data as JSONL, JSON, CSV/TSV, Excel or text; for Excel run `pip install openpyxl` or save as CSV.
Permissions
Runs two bundled Python scripts locally (standard library only) to read your log files and write the dataset into a work folder and an output folder you choose. It makes no network calls, reads no environment variables and never modifies your input files.