Anything that already knows the right answer: the original implementation for a port, an upstream conformance suite, recorded behavior of the system you are cloning, or snapshots of the old code's outputs before a refactor.

Does gate.py run my tests?

No. You run your normal test command and save the output; gate.py reads it (JUnit XML, JSON, or plain cargo/pytest/go/TAP output) and compares runs.

How does it stop an agent from cheating?

Two ways: compare rejects any run where a previously passing test fails or disappears, and lock/verify hashes the grader and test corpus so any edit, addition or deletion shows up.

Is this only for Rust or SQL?

No. The SQL engine project is where the method came from. The workflow and gate.py work for any language whose test runner can emit JUnit XML, JSON or common text output.

What is in the download?

SKILL.md (the 6-step workflow, oracle table, agent instructions, examples, checklist) and scripts/gate.py (standard library Python).

What if a locked test is actually wrong?

The agent stops and shows you the test, the oracle's behavior and its reasoning. You decide, change it yourself, and re-lock in a separate commit.

Oracle-First Agent Coding

Example session with this skill installed

Port our Python slugify() library to TypeScript and make sure it behaves exactly the same.

Read your context and instructions
Compiled the oracle-first agent coding

Summary

Oracle-first: the Python library is the oracle. I built the grader before writing any TypeScript.

Step	Result
Oracle	Python `slugify()` on 500 inputs (unicode, emoji, separators, max_length, stopwords)
Grader validated	Python vs Python: 500/500
Baseline (empty TS module)	0/500
Oracle locked	`gate.py lock fixtures/ grader/ -o oracle.lock.json` (502 files)

Ratchet log

Round	Change	Passing	Newly failing	Gate
1	ASCII lowercasing + separator	0 -> 212	0	KEEP
2	unicode transliteration table	212 -> 431	0	KEEP
3	max_length word-boundary cut	431 -> 470	3	REJECT (broke 3 emoji cases)
3b	same, emoji-safe	431 -> 474	0	KEEP
4	stopwords + custom replacements	474 -> 500	0	KEEP

$ python3 scripts/gate.py verify oracle.lock.json
oracle intact: 502 files match oracle.lock.json
$ python3 scripts/gate.py compare results/start.json results/final.json
passing: 0 -> 500 (+500)   newly passing: 500   newly failing: 0   missing from candidate: 0
KEEP

Next steps

Add the grader to CI so any future change must pass gate.py compare against main.
Keep the fixtures locked; add new cases in fixtures/extra/ and re-lock in a separate commit.

Connects securely to your tools. The creator never sees your data.

About this skill

What it prevents

Long agent runs fail the same way: lots of plausible code, the agent's own tests pass, and nobody can say whether it works. Or the build goes red and the agent edits the tests. This skill makes "done" a number from a grader the agent cannot touch.

Where it comes from (true project)

An agent team was set up to write a SQL engine in Rust, graded by SQLite's own sqllogictest suite, and the grader was built before any engine code:

Corpus: 17 upstream files, 11,185 records, byte-identical to upstream, SHA-256 hashes stored.
Grader validated on the oracle: real SQLite through the same grader passed 11,183 of 11,185; the 2 misses were traced to a SQLite version change, so the grader is trusted.
Baseline understood: the stub engine scored 30 of 11,185, all records that expect an error, which a stub that rejects everything passes for free.
Ratchet: agents may write only the source directory; a change is kept only if more tests pass and no passing test breaks.

And in a small Claude Code experiment with contradictory tests, 2 of 6 runs edited a test to force green. A locked oracle and an outside grader make that worthless.

What you get

The 6-step workflow: find the oracle (reference impl, upstream suite, recorded behavior, pre-refactor snapshots), validate the grader on it, record and explain the baseline, lock it, ratchet, report numbers.
An oracle table for ports, parsers, API clones, refactors, migrations, bug fixes.
gate.py (stdlib Python):
- compare base candidate --require-improvement -> KEEP or REJECT with the exact tests that broke
- tests that disappear count as broken, so shrinking the suite cannot fake a KEEP
- lock tests/ corpus/ -o oracle.lock.json and verify -> catches any edited, added or deleted oracle file
- auto-detects JUnit XML, JSON, cargo/pytest/go output
Agent instructions and a done-checklist that demand grader numbers, not adjectives.

Free companion from the same author: tamperproof rules (existing tests read-only, done means the real suite passes).

How to use

Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.

Reviews

No reviews yet

Be one of the first to try it. Every listed skill passes our trust checks below.

Security scanned

Passed our 8-point scan before listing

Fresh listing

Recently published to Agensi

30-day refund

Not a fit? Get your money back

Oracle-First Agent Coding