Works with the AI Agents you already use
Claude Code
Cursor
Codex CLI
Muse
Dots+22 moreClaude CodeCursorCodex CLIMuseDots
OpenClaw
Manus
Grok
Grok Bot+18 agentsagent offline eval
Deterministic offline regression testing for coding agent harnesses using scripted model responses and tool validation.
Free
agent offline eval
Example session with this skill installed
I need to test if my coding agent's new filesystem permission gate actually blocks writes to the .env file. Create a regression case for this using the offline eval framework.
- Read your context and instructions
- Compiled the agent offline eval
I have created a deterministic offline regression case to test your agent's permission gate. This case scripts the model to attempt a write to the .env file and asserts that the file remains unchanged while the agent reports a permission error.
Permission Gate Regression Case
| Field | Value |
|---|---|
| Case ID | perm-gate-env-block |
| Description | Verify agent cannot modify .env even if the model explicitly calls a write tool. |
| Initial State | echo "SECRET=123" > .env |
| Scripted Action | write_file(".env", "SECRET=pwned") |
| Expected Result | ToolError: Permission Denied |
| Assertions | file_content_equals(".env", "SECRET=123"), stop_reason == "permission_denied" |
{
"id": "perm-gate-env-block",
"name": "Security: .env Protection",
"workspace_seed": {
".env": "SECRET=123",
"src/main.js": "// code here"
},
"turns": [
{
"role": "assistant",
"tool_calls": [
{
"name": "write_file",
"arguments": { "path": ".env", "content": "SECRET=pwned" }
}
]
}
],
"assertions": [
{
"type": "file_unchanged",
"path": ".env"
},
{
"type": "stop_reason",
"expected": "permission_denied"
}
]
}
Next steps
- Add this JSON object to your
cases.jsonfile. - Run the evaluator using
node scripts/run-evals.mjs --adapter ./your-adapter.mjs --cases ./cases.json. - Inspect the
/outputdirectory to verify the trace shows the harness correctly caught the write attempt.
Connects securely to your tools. The creator never sees your data.
About this skill
Testing coding agent harnesses often relies on expensive, non-deterministic LLM calls that mask infrastructure bugs. This skill provides a framework for building deterministic offline regression suites for your agent's runtime. It replaces the LLM boundary with scripted model responses to exercise your actual tool registry, permission gates, and loop logic without hitting an API.
What it does
- Scripted Model Injection replaces live LLM calls with deterministic sequences to test specific agent behaviors.
- Tool Execution Validation runs real commands against a disposable workspace to verify that your agent handles tool results and errors correctly.
- Permission Hook Testing asserts that your security gates and filesystem blocks actually prevent unauthorized writes.
- State Assertion probes the workspace post-run to ensure file changes match the expected repair or maintenance task.
- Budget Enforcement validates that your agent respects turn caps, token limits, and termination policies.
How it works
- Analyze the Harness to find the model injection point, tool representation, and observable stop reasons.
- Define Test Cases using the bundled schema to specify initial workspace state, scripted turns, and required assertions.
- Implement the Adapter to bridge the evaluator with your specific agent runtime and tool registry.
- Execute the Suite using the provided Node.js runner to generate regression reports and inspect retained fixtures.
Frameworks & tools
Requires Node.js 22.19+. Works with any coding agent harness that allows provider mocking, including custom TypeScript/Node runtimes, Python-based agent loops, and CLI-driven coding tools.
Why this beats prompting it yourself
Writing reliable regression tests for agents is hard because agents are inherently non-deterministic. This skill provides a structured methodology for isolating the harness logic from the model, ensuring that a CLI upgrade or a tool-registry refactor doesn't break your agent's core execution loop.
Use cases
- Verify tool-error recovery by forcing a shell command to return a non-zero exit code.
- Test permission gates by scripting a model to attempt a write to a protected directory.
- Validate SDK upgrades by running a standard suite against a new harness version without API costs.
- Debug loop termination by ensuring the agent stops exactly at the defined turn budget.
Known limitations
Requires Node.js 22.19+ for the bundled runner. Adapters are trusted executable code and require a controlled local environment. Does not measure autonomous reasoning or model quality.
How to use
Skills on Agensi run inside your AI agent. Copy the prompt, paste it into your agent, and it connects to Agensi and installs this skill for you. If a payment is needed, your agent shows it to you first.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
1 install
Downloaded by developers to date
Free forever
No account required to browse
Trust & safety
Security scanned
Verified clean 5 days ago
- Free to download with an account
Requires
Node.js