Eval Dataset Preflight: JSONL and Split Checks

Eval dataset: review required

The supplied contract uses exact prompt matching and requires at least one schema-valid example in each of sql and rag. I checked the synthetic eval and train files without changing either file.

Four schema-valid eval rows does not mean four independent, uncontaminated examples. Duplicates and train overlap still need review.

Findings

• Eval line 2 repeats the prompt from eval line 1.
• Eval line 3 repeats the ID from eval line 1, even though its prompt differs. Confirm which ID is intended rather than deleting a row automatically.
• Eval line 5 has a category outside the approved category list.
• Eval line 6 has a non-string prompt. It is a schema error and not usable for prompt matching.
• One unique prompt appears in both splits: eval first line 1 and train first line 1. Eval line 2 repeats that same prompt, so the overlap count is one unique prompt, not one affected eval row.

Category coverage

Schema-valid eval counts are sql: 3 and rag: 1, meeting the selected minimum of 1. These are raw schema-valid counts, not evidence of balanced or representative evaluation coverage. They include duplicate/overlapping examples.

Reproduce

python3 scripts/check_dataset.py --eval fixtures/eval.jsonl --train fixtures/train.jsonl --contract fixtures/contract.json --format json

Exit code: 1, indicating review findings.
Eval SHA-256: 04f05992dae7e1c5982737af6c8f52ffbc303ada890dbdf1e3455e7f012219f3
Train SHA-256: de1a57fff58fee0c20daf6fe850126ae7d46c2370a25efb36ad15f0299f1e3d7

Next steps

Choose the canonical duplicate, fix the ID and schema errors in an approved copy, and decide how to remove the contaminated prompt from the evaluation split without changing the intended test coverage. Then rerun against the reviewed contract.

Only exact configured prompt text was compared. Paraphrase/semantic leakage, answer correctness, nested schemas, label quality and model performance were not checked. No records were deleted, no training or evaluation was run, and no data was uploaded.