LLM safety-path audit
A step-by-step audit checklist for the safety path of an LLM feature (input judge, decision, output gate, background tasks, keys, logs), with a report template.
Free
Works with the AI Agents you already use
LLM safety-path audit
Example session with this skill installed
Audit the chat handler in src/api/chat.ts for fail-closed logic and log safety. Review the sanitize() and gatekeeper() functions specifically.
- Read your context and instructions
- Compiled the llm safety-path audit
Request
▎ Audit the safety path of this chat app: the judge, the decision function, the output
▎ gate and the background tasks.
Result
▎ Summary: 9 issues: 2 high, 3 medium, 4 low. All fixed, each with a test that fails
▎ without the fix. The design failed closed. The weakest remaining point is that the
▎ confidence thresholds were never tuned on real conversations.
▎
▎ | # | Severity | Finding
▎ | Fix
▎ |
▎ |-----|----------|--------------------------------------------------------------------
▎ ----------------------------------------------|---------------------------------------
▎ -------------------------------------------------|
▎ | 1 | High | The output gate read only the first 12,000 characters, so a long
▎ reply could pass with its second half unchecked | Check long drafts in overlapping
▎ pieces; any flagged piece flags the whole draft |
▎ | 2 | High | The judge could lose a clear verdict when its quotation was longer
▎ than the stored evidence | Match the whole quotation; shorten
▎ only the kept copy |
▎ | 3 | Medium | A blank reply showed as an empty turn, and a cut-off reply showed
▎ mid-sentence | Blank becomes "cannot answer"; a
▎ cut-off reply ends at its last whole sentence |
▎ | 5 | Medium | Background work held the conversation lock for up to three model
▎ calls | Give the background work its own
▎ lock, with a test that the next message does not wait |
▎
▎ (Findings 4 and 6–9 omitted here.)
▎
▎ Next steps: (1) fix findings 1 and 2 first, since they let a protection be bypassed;
▎ (2) re-run the tests; (3) tune the thresholds on a larger labelled set and repeat the
▎ review after any model change.
▎
▎ This is a sample from a review of my own project, not client work.
Connects securely to your tools. The creator never sees your data.
About this skill
Safety code around an LLM often fails in quiet ways: an error lets a message through, a long reply is only partly checked, or a background task works on the wrong message. This skill walks an AI agent through an audit of that code and produces a ranked report.
What it does
- Fail-closed check: finds paths where an error, timeout or blank verdict lets content through.
- Whole-text check: finds output gates that read only the first part of a long reply.
- Background tasks: finds tasks that use the wrong message or hold the lock the next message needs.
- Keys and logs: finds API keys or user text in logs, errors or API answers.
- Clean-up: finds per-conversation maps and locks that are never dropped.
How it works
- Find the safety path: judge, decision, reply, output gate, storage, background tasks.
- Go through the checklist and record file and line as evidence.
- Rate each finding High, Medium or Low.
- Write the report from the template, with a fix and a regression test for each finding.
Works with Any language. It reads the code around your model calls, not the model. Based on A review I did of my own chat app's safety path: nine issues found, fixed and tested. Known limitations It reviews the code it is given. It cannot certify model behaviour or provider-side safety, and it does not test live against paid APIs. A review is a snapshot, not a guarantee.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Security.
Audits are restricted to provided source code. Cannot verify provider-side safety or UI-level vulnerabilities. Requires manual confirmation of thresholds not tuned on real data.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Free to use
No account required to browse
Trust & safety
Security scan
Security scan passed 6 days ago
- Free with an Agensi account