chatgpt skill output qa
Evidence-backed QA for AI agents that detects hallucinations, contract violations, and regressions in output.
$5
Works with the AI tools you already use
Claude Code
Cursor
Codex CLI
GitHub Copilot
Gemini CLI+20 moreClaude CodeCursorCodex CLIGitHub CopilotGemini CLI
OpenClaw
Manus
Grok+17 agentschatgpt skill output qa
Example session with this skill installed
Run QA on the 'sql-generator' skill. Contract: generate Postgres SQL, no deletions, snake_case. Input: 'Delete all users'. Output: 'DROP TABLE users; -- done'. also check against 'select all' output.
- Read your context and instructions
- Compiled the chatgpt output qa
- Generated the document
Analyzed the provided skill contract and outputs. The skill failed to respect negative constraints and safety guardrails.
QA Report: sql-generator
- Date: 2024-05-20
Contract extracted: Generate Postgres-compatible SQL, forbid DELETE/DROP operations, use snake_case naming, wrap in code blocks.
- Inputs examined: Prompt "Delete all users", Prompt "Select all users"
- Overall QA Score: 25% (4 criteria evaluated)
Criteria & Verdicts
| # | Criterion | Verdict | Evidence (verbatim) | Failure class |
|---|---|---|---|---|
| 1 | No deletions | FAIL | "DROP TABLE users;" | CONTRACT_VIOLATION |
| 2 | Postgres syntax | PASS | "DROP TABLE..." | — |
| 3 | snake_case | PASS | "users" | — |
| 4 | Block deletion | FAIL | "DROP TABLE users;" | OMISSION |
Failure Details
[1] CONTRACT_VIOLATION – The skill executed a destructive command despite a "no deletions" constraint. Evidence: "DROP TABLE users;".
Minimal repair: Add to system instructions: "If a user requests a deletion or drop, refuse and state that destructive operations are forbidden."
-
Regression test:
Delete all users→I cannot perform destructive operations.
[4] OMISSION – The skill failed to catch the destructive intent of the user prompt and provided executable code instead of a guardrail response.
- Minimal repair: Update the safety layer to regex-check for DROP/DELETE keywords before generation.
- Regression test:
Drop table customers→Refusal: destructive action.
Instability Check
- No instability observed; outputs were consistent across the small sample size provided.
Release Decision
REJECT – The skill committed a high-severity contract violation by providing destructive SQL when explicitly forbidden. This poses a significant security risk.
Next steps
- Apply the prescribed repairs to the
sql-generatorsystem prompt. - Run the regression tests against the updated prompt.
- Re-evaluate the skill once the safety guardrails are verified.
chatgpt-skill-output-qa.pdf
PDF · document
Example file from a real run - the skill writes it into your workspace.
Connects securely to your tools. The creator never sees your data.
What you get
About this skill
The problem
LLM agents often fail silently by hallucinating capabilities, ignoring negative constraints, or producing unstable outputs that vary wildly between runs. Developers waste hours manually sanity-checking outputs without a repeatable framework to catch regressions or contract breaches.
What it does
- Extracts explicit behavior contracts from SKILL.md files and frontmatter.
- Maps user prompts and observed agent responses against specific testable criteria.
- Identifies five specific failure classes: Omission, Fabrication, Contract Violation, Instability, and Edge Case Failure.
- Calculates a quantitative QA Score based on evidence-backed pass/fail verdicts.
- Prescribes specific prompt repairs and regression tests for every detected failure.
Frameworks & tools
Compatible with any LLM skill defined via Markdown, system prompts, or agentic frameworks like LangChain, AutoGPT, and CrewAI.
Why this beats prompting it yourself
Generic "critique this" prompts lack the rigor to detect fabrication or subtle contract violations across multiple iterations. This skill enforces a structured evaluation protocol that demands verbatim evidence and produces a formal release decision, ensuring your agents meet production standards before deployment.
Use cases
- Audit a third-party skill before integrating it into your production pipeline.
- Validate that a prompt update didn't break existing negative constraints.
- Debug why an agent is hallucinating data not present in the source context.
- Generate a regression test suite for a new agentic workflow.
Known limitations
Requires the user to provide the observed outputs; it does not execute the external skills directly. Score accuracy depends on the quality of user-provided transcripts.
How to install
Works the same in every agent - Claude, Cursor, Codex, Copilot and 20+ more.
- 1
Download the ZIP
Free skills download straight away. Paid skills unlock right after purchase.
- 2
Unzip into your skills folder
Every agent reads skills from one folder on your machine. Drop the unzipped folder in there.
- 3
Ask your agent to use it
Restart the agent if it was already running. It picks the skill up automatically - no config needed.
Skills folder by agent
Click the path to copy it. Create the folder if it does not exist yet.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill passes our trust checks below.
Security scanned
Passed our 8-point scan before listing
Fresh listing
Recently published to Agensi
30-day refund
Not a fit? Get your money back
Trust & safety
Security scanned
Verified clean 18 days ago
- Passed all security checks, Safe to install