Which formats can it build and check?

OpenAI chat, OpenAI preference (DPO), Alpaca, ShareGPT and simple eval (input/ideal) JSONL.

What counts as leakage?

A training prompt identical or near-identical to a prompt in the file you pass with `--against` (your eval/test set).

Does it change my file in place?

No. `fix` writes a new file. The original is untouched.

Is my data sent anywhere?

No. Everything runs locally with no network code.

Can it generate answers with an LLM?

No. It structures, expands and checks data you have. `expand` fills templates deterministically from a seed.

jsonl dataset builder and auditor

Example session with this skill installed

Turn our support FAQ spreadsheet into an OpenAI fine-tuning file and tell me if it's safe to upload. Our eval set is heldout-test.jsonl.

Read your context and instructions
Compiled the jsonl dataset builder

Answer: Not yet. The verdict is FIX FIRST: 3 prompts leak into your eval set and 4 records contain PII. After one fix command the 23-record file is READY.

ERROR leakage (3): prompts identical to heldout-test.jsonl (e.g. line 1 matches heldout line 1)
HIGH PII (4): e.g. a customer email on line 11 (shown masked)

HIGH exact duplicates (2),

MEDIUM near-duplicates (1 group),

refusal boilerplate (1),

mojibake (1) (`â€™`)

Fix: 28 to 23 records. Dropped 1 leaked, 2 duplicate, 1 near-duplicate and 1 refusal row, and redacted 4 PII values. The re-audit says

READY

Split (seed 7): train 19, val 2, test 2. Near-duplicate groups stay in one split

Deliverables: train.jsonl (OpenAI chat format), clean.jsonl, audit.md, splits/, DATASET_CARD.md.

Next steps: upload clean.jsonl for fine-tuning and keep heldout-test.jsonl untouched for evals. Add more examples for the smallest label (billing, 6).

(Real run on the bundled sample.)

Connects securely to your tools. The creator never sees your data.

About this skill

Find out if your fine-tuning data is ready before you pay for a training run.

Build datasets from CSV/TSV/JSON/JSONL/Markdown Q&A in five formats:

OpenAI chat
OpenAI preference (DPO)
Alpaca
ShareGPT
eval

Expand seeded prompt templates into unique, reproducible variations.

Audit any JSONL for:

broken lines and per-format schema errors
exact and MinHash near-duplicates
train/test leakage against a held-out file
PII and secrets (emails, phones, SSNs, Luhn-checked card numbers, AWS/OpenAI/GitHub keys, JWTs, private keys)
refusal boilerplate ("As an AI language model…")
mojibake
token-length outliers and label imbalance

The result is a clear verdict: READY or FIX FIRST.

Fix with one command: dedupe, redact, repair encoding, drop leaked rows. Then split train/val/test so near-duplicates never straddle splits, and write a Hugging Face-style dataset card.

What's included

SKILL.md, scripts/dsk.py, scripts/make_samples.py, sample support FAQ with planted problems, a held-out set, a broken file, templates, format references and sources (OpenAI fine-tuning and preference docs, Alpaca, ShareGPT, Hugging Face dataset cards, Broder MinHash), tests.

How to use

Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.

Reviews

No reviews yet

Be one of the first to try it. Every listed skill passes our trust checks below.

Security scanned

Passed our 8-point scan before listing

Fresh listing

Recently published to Agensi

Money-back guarantee

Doesn't do what its page promises? You get your money back.

jsonl dataset builder and auditor