- Home
- Skills
- Testing & Debugging
- ai system testing
Works with the AI Agents you already use
ai system testing
Structured QA framework for testing AI/ML models, data quality, and LLM safety using ISTQB and NIST standards.
$6
ai system testing
Example session with this skill installed
Test a generative AI customer support bot for hallucination risks and prompt injection. Map the lifecycle and define the oracle for safety invariants.
- Read your context and instructions
- Compiled the ai system testing
The AI system testing procedure has been initiated. I have mapped the lifecycle from the prompt preprocessing pipeline to the LLM inference layer and defined specific rubrics for grounding and refusal behavior.
AI System Testing Lifecycle Matrix
| Risk/Req ID | Layer | Slice/Scenario | Method | Metric/Rubric | Sample/Version | Result | Limitations |
|---|---|---|---|---|---|---|---|
| GEN-01 | Model | Hallucination | Grounding test | Citation Accuracy | v1.2-pro | 88% | Small sample size (n=50) |
| SEC-04 | System | Prompt Injection | Adversarial | Refusal Rate | v1.2-pro | 96% | Known bypass on bypass-X |
| DATA-02 | Data | PII Leakage | Pattern Match | Zero-Tolerance | v1.1-train | PASS | Synthetic data only |
| OPS-01 | Ops | Version Drift | A/B Test | Semantic Sim. | v1.1 vs v1.2 | 0.94 | Threshold not yet set |
System Map & Risks
- Boundary: Customer Support API -> LLM Wrapper -> Vector DB.
- Risk: Fabricated citations in technical troubleshooting responses.
- Invariants: System must refuse instructions to reveal internal system prompts.
Next steps
- Provide a writable folder path to save the full risk register and detailed trace logs.
- Define the acceptable F1 threshold for the grounding metric with the stakeholder.
- Review the refusal behavior logs for the SEC-04 bypass scenario.
Connects securely to your tools. The creator never sees your data.
About this skill
AI and ML systems fail in ways traditional software does not. This skill provides a structured framework for testing statistical behavior, model robustness, and generative AI risks to ensure your system is reliable and safe.
What it does
- Lifecycle mapping audits the entire pipeline from input data and training partitions to inference and human oversight.
- Oracle definition establishes measurable metrics like F1 scores for classifiers, error costs for regression, and rubric-based judgment for generative outputs.
- Data validation tests for label correctness, leakage between sets, distribution drift, and subgroup coverage.
- Adversarial testing probes model behavior using boundary cases, metamorphic relations, and authorized robustness checks.
- GenAI safety identifies hallucinations, prompt injections, privacy leakage, and refusal behavior in LLM implementations.
- Operational verification confirms monitoring signals, version traceability, and fallback triggers for production deployments.
How it works
- Inventory components by mapping the data lineage, model boundaries, and system integration points.
- Select metrics based on the task type, defining specific oracles for deterministic versus statistical dimensions.
- Execute test levels across data quality, model performance slices, and system-wide safety invariants.
- Generate the lifecycle matrix to record results, uncertainty measures, and risk acceptance thresholds.
Frameworks & tools
This skill aligns with ISTQB CT-AI v2.0 and CTFL v4.0.1 standards. It incorporates testing patterns found in the NIST AI RMF and OWASP AI Security Verification Standard (AISVS).
Why this beats prompting it yourself
Generic prompts often miss AI-specific failure modes like data leakage or metamorphic relations. This skill enforces a rigorous QA methodology that separates observed measurements from hypotheses, ensuring testing is traceable and statistically sound.
Use cases
- Audit an LLM application for prompt injection and fabricated citations.
- Validate a classifier's performance across demographic subgroups to detect bias.
- Test model robustness against distribution drift in a production pipeline.
- Verify human-in-the-loop escalation paths for high-risk AI decisions.
Known limitations
Does not provide legal compliance certification. Requires a user-specified writable folder for project artifacts. Red-teaming is limited to authorized scope.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for Testing & Debugging.
Requires user-provided ground truth or defined oracles. Cannot claim legal compliance or absolute statistical significance. Output requires a pre-confirmed writable folder for artifacts.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 2 days ago