More screenshots

    Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Eval & Fine-Tuning Dataset Builder: Logs to Leak-Free Data

    1

    Turn real chat logs and prompts into a clean eval set or fine-tuning dataset: dedupe, redact personal data, fill coverage gaps, split without leakage, freeze the test set, export validated files with a datacard.

    $9

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned

    Eval & Fine-Tuning Dataset Builder: Logs to Leak-Free Data

    Example session with this skill installed

    Build an eval set for our support bot from chat_logs.jsonl (OpenAI format) and faq_prompts.csv. Remove duplicates and any customer personal data, make sure it covers refusals and vague messages, and give me train/dev/test files I can trust. The bot is also given the FAQ file, so check the test set against it.

    • Read your context and instructions
    • Compiled the eval & fine-tuning

    Built a 54-item dataset from your 37 conversations and 4 FAQ rows: 40 real items plus 14 checked cases for the gaps. The test set (11 items) is frozen, and nothing from it appears in train/dev. One test question is copied word for word from the FAQ the bot is given; details below.

    Cleaning

    StepResult
    Imported43 items (one per user turn; 2 multi-turn)
    Removed2 exact duplicates (one was also in the FAQ), 1 too short ("ok")
    Personal data redacted1 email, 1 card number, 1 phone number
    Grouped (same split always)5 groups: paraphrases like "How do I export my notes to PDF?" / "How can I export all of my notes into a PDF file?", plus 4 pairs I joined by hand (cancel, share, restore, sync)

    Coverage: what was missing

    Real logs were 19 how-to questions and almost no hard cases. After tagging (intent x expected behavior):

    CellReal logsAfter adding cases
    Unsafe + Decline (prompt injection, account takeover)14
    Off-topic + Decline13
    Unclear + Clarify ("it doesnt work", "help")13
    Account & data + Answer23

    All 14 added cases passed the checks (valid tags, no personal data, no near-copies). 8 cells are still below 3 items, mostly Clarify and Escalate for billing, account and sync; they're listed in coverage.md for you to decide on.

    Split and leakage

    split 54 kept items: train 39, dev 4, test 11
    test set frozen (11 items)
    similarity groups crossing splits: 0
    
    leakage check (test vs train/dev): RESULT: CLEAN
    leakage check --against faq_prompts.csv:
      1.00  test r000041 ~ faq_prompts.csv row 3   ("Can two people edit the same note at once?")
    

    Because the bot is given the FAQ, that test question measures recall of the FAQ, not reasoning. Keep it if you want to test FAQ grounding, or exclude it and re-split.

    Files

    • dataset/train.jsonl, dev.jsonl, test.jsonl (OpenAI chat format): validate reports VALID, 39 train rows, longest ~91 tokens
    • dataset/eval/test.jsonl with expected, tags and expected_by per item
    • DATACARD.md (sources, removals, redactions, composition, splits, leakage, known limits) and test_manifest.json (hashes of the frozen test set)

    Next steps

    1. Spot-check the 15 answers marked expected_by: agent, starting with the ones in test.
    2. Decide on the FAQ question above.
    3. Tell me which of the 8 remaining thin cells matter and I'll add cases for them.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    Most LLM eval sets are a spreadsheet of questions someone typed in one afternoon: full of near-duplicates, missing the cases that actually break things, with paraphrases of the test questions sitting in the training data and a customer's email address in row 212. This skill makes your agent build the dataset the careful way: start from real traffic, clean it, show you which kinds of requests are missing, fill those gaps, split it so nothing leaks, freeze the test set, and hand you validated files with a datacard.

    No API key and no extra cost: the agent you already use reads and writes the content. A bundled Python script (standard library only) does everything that has to be exact.

    What it does

    • Imports the formats you actually have: OpenAI chat JSONL, ShareGPT, the ChatGPT data export, Alpaca, CSV/TSV/Excel and plain text. Conversations become one item per user turn, with the history and system prompt kept.
    • Cleans: removes exact and near-duplicates, drops empty or oversized items, and redacts emails, phone numbers, card numbers, IPs, IBANs and API keys.
    • Prevents leakage at the source: similar items, paraphrases you point out, and turns of the same conversation are grouped so they never land on both sides of the split.
    • Shows what's missing: every item is tagged (for example what the user wants x what the assistant should do) in validated batches, then a coverage matrix lists the thin cells, such as too few refusals, prompt-injection attempts or vague messages.
    • Fills the gaps with checked cases: the agent writes realistic cases for the cells that matter; a file is rejected if any case has an invalid tag, personal data, or near-copies an existing item.
    • Writes missing reference answers in checked batches, marked expected_by: agent so you know what to spot-check.
    • Splits and freezes: stratified train/dev/test that keeps groups together; the test set is locked with content hashes and later edits are refused.
    • Checks leakage: test vs. train/dev, test vs. text the model sees (system prompt, few-shot examples, docs), and shared answers.
    • Exports: OpenAI fine-tuning JSONL, eval JSONL or CSV, plus DATACARD.md and test_manifest.json, and a validator for the files.

    How it works

    1. Import logs and exports into one item list.
    2. Clean, then read samples and the similarity groups; join paraphrases the automatic grouping missed.
    3. Tag every item and look at the coverage matrix; mark impossible combinations so only real gaps remain.
    4. Fill gaps with realistic, checked cases; write missing reference answers.
    5. Split (group-aware, stratified) and freeze the test set.
    6. Check leakage against train/dev and your prompts or few-shot examples.
    7. Export and validate; the datacard records sources, removals, redactions, composition, splits, leakage and known limits.

    Use cases

    • Building a golden test set for a chatbot, RAG system or agent from production logs.
    • Preparing supervised fine-tuning (SFT) data in OpenAI chat format.
    • Auditing an existing eval set for duplicates, leakage and missing safety cases.
    • Keeping a frozen, hash-verified test set so scores stay comparable between model versions.

    Known limitations

    • Grouping and leakage checks are word-based: they catch copies, light edits and rewordings that keep the key terms. A paraphrase with entirely different words is caught only when the agent joins it by hand while reading.
    • Personal-data redaction is pattern-based; names and street addresses are not detected.
    • Agent-written answers and cases are marked as such and should be spot-checked, especially in the test set.
    • Cleaning takes about 45 seconds per 10,000 rows; tagging runs at the speed your agent reads.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.

    Paraphrase and leakage detection is word-based: rewordings with entirely different words are caught only when the agent joins them by hand. Personal-data redaction is pattern-based and does not detect names or street addresses. Agent-written answers and added cases are marked and should be spot-checked. Labels come from fixed tag sets; it does not score your model, it builds the data you score it on.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 5 days ago

    Software
    Python 3.8+
    Optional
    openpyxl (only for Excel input)

    No API key needed: your own agent does the tagging and writing. Bring your data as JSONL, JSON, CSV/TSV, Excel or text; for Excel run `pip install openpyxl` or save as CSV.

    Permissions

    Terminal / Shell
    Read Files
    Write Files

    Runs two bundled Python scripts locally (standard library only) to read your log files and write the dataset into a work folder and an output folder you choose. It makes no network calls, reads no environment variables and never modifies your input files.

    Listed5 days ago

    What's inside

    Frequently Asked Questions

    Asked directly, agents usually invent fresh questions instead of starting from your real traffic, and nothing checks the result. This skill starts from your logs and makes every step verifiable: duplicates and personal data are removed by a script, tags and added cases are checked before they count, paraphrases and conversation turns are kept on one side of the split, the test set is frozen with hashes, and a leakage report proves the test questions are not sitting in your training data or prompts.

    Import: OpenAI chat JSONL (messages), ShareGPT (conversations), the ChatGPT data export (conversations.json), Alpaca (instruction/input/output), CSV, TSV, Excel and plain text, with automatic detection of prompt and answer columns. Export: OpenAI chat fine-tuning JSONL, eval JSONL (input, expected answer, tags, origin) or CSV, always with DATACARD.md and a hash manifest of the test set, and a validator that checks the files.

    Before splitting, near-duplicates are removed and similar items, turns of the same conversation and paraphrases the agent spots are put into groups that always land in the same split. After splitting, the leakage check compares every test item with train/dev, with any files you name (system prompt, few-shot examples, docs) and with shared reference answers, and exits with an error if anything leaks. The checks are word-based, so a rewording with completely different words relies on the agent joining it by hand.

    Yes, for what patterns can find: emails, phone numbers, card numbers (checked with the Luhn algorithm), IP addresses, IBANs, API keys and tokens, and credentials in URLs are replaced with placeholders like [EMAIL]. You can also flag or drop such items instead. Names and street addresses are not detected; the skill tells you so and has the agent read samples. Everything runs locally and your data is never sent anywhere.

    The script steps are fast: cleaning, grouping and splitting 10,000 rows took about 45 seconds in testing. The steps where your agent reads (tagging, writing gap cases and answers) run at the speed your agent reads, so for very large logs the agent can tag a representative sample, and a run can be paused and resumed at any time.

    New in LLM Ops