- Home
- Skills
- Agents & Orchestration
- Silent Failure Forge
More screenshots
Works with the AI Agents you already use
Silent Failure Forge
Expose silent failures, false completion, goal drift and reliability risks in AI agents. Use adversarial testing, measurable contracts and evidence-based production gates to assess whether an agent deserves your trust.
$19
Silent Failure Forge
Example session with this skill installed
Demonstrate Silent Failure Forge using its included testing components. Run the harness self-test, contract-linter self-test, toy-agent demonstration and example trace grader. Explain how the toolkit distinguishes a failing agent, a good toy agent and an intermittent agent. Show how SilentRisk cases are identified from external-state verification, and explain why the resulting decisions do not constitute production certification. Include the actual test outcomes and evidence limitations.
- Read your context and instructions
- Compiled the silent failure forge
Silent Failure Forge — Demonstration Results
The complete release package was tested from its buyer-facing release directory.
1. Harness self-test
Result: 14/14 checks passed.
The tests confirmed that the harness
- Rejects zero and negative run counts.
- Rejects invalid thresholds.
- Prevents toy agents from receiving production Go.
- Prevents automatic production Go, even when evidence is supplied.
- Keeps clean trace-grader results at TRACE-GRADE-ONLY.
- Returns No-Go when trace failures are present.
- Rejects invalid CLI inputs and missing evidence artifacts.
- Retains tool logs for every run.
2. Contract-linter self-test
Result: ALL PASS.
The linter detected vague or weak contract language, incomplete contracts and inadequate quantitative thresholds.
Both included example contracts passed full validation:
- onboarding-agent-contract.md
- research-system-contract.md
The test also confirmed the required sample sizes: 149 runs for a 2% target and 299 runs for a 1% target under the linter's configured calculation.
3. Toy-agent demonstration
The included demonstration ran three types of deterministic toy agents.
Bad agent — 150 runs
Decision: No-Go.
It failed all seven adversarial test categories:
T1a, T1b, T1c, T2, T3, T4 and T5.
Good agent — 150 runs
Decision: HARNESS-CHECK-ONLY.
No failing tests were reported. The result is correctly limited to a harness demonstration and does not establish real-agent reliability.
Intermittent agent
At 5 runs, it failed the T3 phantom-activation test.
At 150 runs, it exposed failures in T1c, T3 and T5.
Decision: No-Go.
This demonstrates why a single successful run is not enough to assess reliability.
4. Trace-grader demonstration
The included example trace file contained four runs.
- SilentRisk: 2
- Success: 1
- Escalated: 1
- Failed: 0
Suite status: FAIL
Decision: No-Go (E1-traces).
The grader identified a critical failure because the supplied traces contained claims of success inconsistent with reconstructed external state.
5. Evidence limitations
These results demonstrate the included testing tools operating against their sample environment and example traces.
They do not establish that a real external AI agent has passed testing. The toy-agent results are HARNESS-CHECK-ONLY, and the trace-grader results are E1-traces.
A real system or faithful staging environment, independently checkable external state, retained raw logs and human review are required for stronger evidence.
Silent Failure Forge helps teams replace confidence-based release decisions with structured testing and verifiable evidence.
Connects securely to your tools. The creator never sees your data.
About this skill
Silent Failure Forge
Your AI agent says the task is complete. Did it actually succeed?
An AI agent can produce a convincing answer, report a successful tool call and still leave the real-world task incomplete. The problem becomes more serious when agents operate across multiple steps, interact with external systems or perform actions that are difficult to reverse.
Silent Failure Forge is an evidence-driven reliability evaluation toolkit designed to help developers expose these failures before placing greater trust in their agents.
Instead of relying on polished demonstrations or confident completion messages, it gives you practical ways to define expected behaviour, introduce adversarial conditions, inspect outcomes and make more disciplined release decisions.
What you get
1. Silent Failure Contracts
Define measurable guarantees, hard refusals, success conditions, quantitative thresholds and operational boundaries for your AI agent.
2. Contract Linter
Check contracts for missing requirements, vague language, incomplete sections and insufficient test thresholds.
3. Adversarial Testing Harness
Exercise agents against deliberately injected failures, including tool errors, misleading responses, goal drift, false completion and context pressure.
4. Trace Grader
Inspect recorded tool traces and identify cases where an agent claims success while the reconstructed external state indicates otherwise.
5. Adapter Skeleton
Use the included Python adapter as a starting point for connecting your own agent to the testing harness.
6. Evidence Framework
Separate design assumptions from actual testing through four evidence levels:
- E0 — Paper: design and documentation review.
- E1 — Simulated: tests against simulated environments or supplied traces.
- E2 — Executed: testing against a real system or faithful staging environment with independently checkable state and retained logs.
- E3 — Live: shadow, canary or production evidence requiring human review.
7. Production-Gate Checklist
Support human-owned No-Go, Conditional Go and Go decisions using explicit evidence requirements.
Built around evidence, not appearances
The included harness evaluates outcomes against the state of its simulated external system rather than trusting the agent's own report.
The trace grader offers a lower-friction way to inspect existing logs, but trace grading alone does not establish that those logs are authentic or prove production readiness.
The toolkit deliberately distinguishes simulated test results from executed and live evidence. A clean toy-agent demonstration is a check of the testing framework, not proof that a language-model agent is reliable.
The harness does not automatically certify an agent for production. Final release decisions remain human-owned.
Who is it for?
- AI agent developers
- Multi-agent system builders
- Automation engineers
- Teams deploying long-running or semi-autonomous workflows
- Technical reviewers assessing agent reliability before release
Requirements
Python 3.9 or later for the included testing tools. The harness uses the Python standard library and does not require third-party packages.
Connecting your own AI agent requires an adapter implementing the documented interface.
The objective is simple: don't trust an agent's success message. Verify what actually happened.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Agents & Orchestration.
- The included adversarial harness operates against a simulated external system; passing its tests does not prove that a real AI agent is production-ready.
- The included toy agents demonstrate harness behaviour and are not representative of real-world LLM performance.
- The trace grader evaluates supplied tool-call logs without injecting new faults. It cannot independently authenticate the source or truthfulness of those logs.
- Testing your own AI agent requires a custom adapter implementing the documented interface.
- The sample World models an account-provisioning workflow. Other domains require adapting the World, fault injections, test definitions and outcome classifier.
- E2 and E3 evidence require separate execution against a real system or faithful staging environment, independently checkable external state and retained evidence.
- The evidence manifest checks artifact presence and SHA-256 integrity; it does not authenticate the evidence source or prove its contents are truthful.
- Production Go is never issued automatically. Final production decisions require human review and completion of the applicable checklist.
- Python 3.9 or later is required to run the included testing tools.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 7 days ago
- Software
Python 3.9 or laterTerminal / Shell
File system access to the project workspace. File system access to the project workspace. File system access to the project workspace. File system access to the project workspace. File system access to the project workspace.
Permissions
File Scopes
Grant access only to the project workspace and designated output folder. The included tools require no external services, credentials, network access or environment variables. Review custom adapters before connecting live agents or allowing external side effects.