Pre-Declared ML Experiments and Certified Decision Gates
This kit is the paperwork and the arithmetic that close both: declare, run, read once, adopt or stop.
$35
Works with the AI Agents you already use
Pre-Declared ML Experiments and Certified Decision Gates
Example session with this skill installed
Our re-ranker picks a product match when the inlier count is over 20, but that number was tuned by eye. Give me a certified threshold with a real error bound, and a way to stop my team from moving the goalposts after each run.
- Read your context and instructions
- Compiled the pre-declared ml experiments
Summary
Certified an inlier floor of 40 (not 20) at 5% target risk, 95% confidence, on a one-shot read of the test half; and put the predeclaration and result paperwork in front of every future run.
Certification
python scripts/experiment_gate.py certify --rows rows.jsonl --ladder 20,30,40,50,60,80,100 --target-risk 0.05 --confidence 0.95 --min-coverage 0.20 --lock outputs/v12-test-read.lock.json --output outputs/v12-summary.json
| Rung | Released (calibration) | Errors | Bonferroni bound |
|---|---|---|---|
| 20 | 1,102 | 128 | 0.134 |
| 30 | 868 | 61 | 0.089 |
| 40 | 701 | 22 | 0.047 |
| 50 | 590 | 14 | 0.040 |
Loosest passing rung
40. Test half read once at 40: 688 released of 1,497 kept (coverage 0.46), 19 errors, 95% bound 0.041.
Adopted. The lock file now refuses any second read with another floor, bar or set (exit 2).
Paperwork
docs/V12_PREDECLARATION.mdcommitted before the run (hash3f9c1a2): the change, two arms on the frozen 3,004-photo set, bar "+3 points, at most half as many to-wrong as to-right", four predictions.docs/V12_RESULT.md: cites3f9c1a2, verdicts on all four predictions (two wrong), "Not adopted" for arm B, "Nothing since the declaration" in the code section.python scripts/experiment_gate.py check-docs --predeclaration ... --result ...-> ok.
Tests
python -m unittest test_experiment_gate -v: 10 OK.
Connects securely to your tools. The creator never sees your data.
About this skill
After enough experiments every team discovers the same two ways to lie to itself: tuning a threshold on the set that then confirms it, and writing the bar after seeing the result. This kit is the paperwork and the arithmetic that close both: declare, run, read once, adopt or stop. It comes from a run of more than a hundred numbered experiments on a product-identification model, where the rules below were each learned by breaking them.
Paperwork (templates/): a predeclaration (the change, the arms on one frozen set, the bar as numbers, predictions that can be wrong) committed before any arm runs; a result that cites the predeclaration's hash in its first paragraph, gives every prediction a verdict including the wrong ones, says adopted or not, and discloses what changed in the code since the declaration. check-docs fails a result that skips any of that.
Arithmetic (experiment_gate.py, standard library): a binomial upper bound by bisection (tested against the closed form and the exact sum); a fixed hash split of query ids into calibration and test halves; a ladder of floors walked on the calibration half with the 1 - confidence split across the ladder (Bonferroni), the loosest rung at or under the target risk chosen; one read of the test half at that floor, enforced by a lock file that refuses a second read with a different floor, bar or set; adopted only when the test bound and coverage clear the bar; counts-only JSON summaries; exit 0 adopted, 1 not, 2 refused.
Rules (reference/rules.md): hard boundaries (frozen holdouts, no same-image validation, versioned annotations, no data in Git), the eight selection conditions and when to reject, certified-floor method, seven common bugs with fixes, the before-and-after reproducibility checklist, what not to do next.
10 offline tests. Python 3.9+, no packages, no network.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Requires independent queries. Floors outside the pre-defined ladder are not certified. The checker validates structure and presence of sections, not the scientific quality of the content.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 19 days ago
- Software
Python 3.9+
GitYour model and evaluation set
Export them as rows.jsonl with id, score and correct fields.
Permissions
Shell to run the Python script and tests; read files for rows.jsonl and the two documents; write files for the summary JSON and the test-read lock file. No network access, no environment variables.