AI Agent Testing and Evaluation Bundle

About this skill

What's included (6 skills)

• agent-run-diff

  Compare baseline and candidate AI agent run logs, event streams, receipts, or exported bundles to identify added, removed, reordered, or changed tool calls, outputs, errors, state events, and timings. Use when the user asks what changed between agent runs or whether behavior shifted after a prompt, model, tool, policy, or code update. Reports evidence without inferring hidden reasoning or unsupported causality.

• agent-run-regression-builder

  Turn observable AI agent logs, checkpoints, tool events, errors, or receipts into bounded incident bundles, replay-risk plans, and regression fixtures. Use when the user asks to debug a failed tool-using agent run, reproduce an incident from recorded evidence, or create a regression test from a trace. Does not recover hidden reasoning or execute recorded side effects.

• dataset-provenance-gate

  Curate externally sourced AI datasets into traceable release packages with source pins, rights-evidence holds, transformation lineage, deduplication, split-leakage checks, attribution, and quality reports. Use for commercial or public dataset packaging; it organizes evidence and does not provide legal clearance.

• retained-proof

  Write an audit or verification report that keeps its proof — including the runs that failed — in a form a stranger can re-check: a scope-and-claim boundary up front, a finding ledger with IDs and dispositions, an evidence manifest (path, bytes, SHA-256 per file, with its own hash quoted), a negative control proving the verifier can fail, and claim language that never outruns what was run. Use this whenever you are asked to audit, verify, certify, review, test, or "prove" anything about code, a dataset, a runtime, an agent, or a build — and whenever a report, receipt, or write-up will be read by someone deciding whether to trust the work. Use it even if the user only says "write up the results" or "is this safe to ship": a report without retained proof is an opinion.

• sealed-eval-pack-builder

  Build or review deterministic, blind, provenance-bound AI evaluation packs with locally recomputable labels, sealed answers, integrity manifests, and bounded result claims. Use for release-grade classification or policy eval datasets; do not use when subjective LLM-as-judge scoring is the intended ground truth.

• untrusted-content-snapshot

  Capture and statically inspect untrusted web pages, documents, messages, retrieved context, or tool descriptions as point-in-time evidence snapshots before an AI uses them. Use when the user asks to preserve external content, separate evidence from embedded requests, retain source provenance, or prepare bounded AI analysis. Does not determine that content is safe or trustworthy.