AI Agent Execution Auditor

Stop guessing if your AI agents actually finished their tasks. When an agent claims a workflow is complete, you need to know whether the underlying tool calls, API responses, and execution logs actually support that claim.

AI Agent Execution Auditor examines agent execution traces and separates recorded evidence from unsupported completion claims.

What it does

• Verifies tool calls by matching agent claims against recorded API results, responses, and status information.

• Identifies execution gaps between the original task requirements and the actions that were actually performed.

• Detects contradictions where an agent reports success despite receiving tool errors, failed actions, or timeouts.

• Maps evidence to specific task requirements using four clear statuses: Verified, Partially Verified, Unverified, and Contradicted.

• Surfaces operational risks, including exposed credentials, unauthorized destructive actions, and other potentially significant issues found in execution logs.

How it works

Extract the task requirements by identifying the objectives, constraints, required actions, and success conditions from the initial prompt or workflow definition. Reconstruct the execution timeline by organizing recorded intents, tool calls, actions, and results into a chronological sequence. Audit completion claims against the available evidence and determine whether each material claim is supported, partially supported, unsupported, or contradicted. Generate a structured audit report with a requirement-to-evidence matrix, prioritized findings, completion assessment, evidence gaps, and recommended remediation.

Frameworks and tools

Works with agentic systems that export execution logs or chat histories, including LangGraph, CrewAI, AutoGen, and custom implementations using JSON, JSONL, Markdown, or plain-text transcripts.

Why this beats prompting it yourself

Generic prompts can fall into the same trap as the agents they are supposed to evaluate: accepting “I did it” at face value.

This skill applies a structured, evidence-based audit methodology. It treats execution records as evidence to be analyzed rather than instructions to be blindly followed, making it easier to distinguish actual execution from unsupported claims of completion.

Use cases

• QA for agentic workflows — Evaluate whether production agents actually completed the required actions.

• Incident response — Investigate failed automated workflows and identify the exact execution or tool-call failures.

• Benchmark verification — Confirm whether an agent followed specific requirements and constraints during an evaluation run.

• Agent reliability reviews — Identify recurring execution gaps, unsupported claims, and evidence weaknesses.

• Workflow monitoring — Review multi-step agent runs before considering them fully completed.

Known limitations

The skill does not independently access external systems or verify real-world outcomes. Audit conclusions are strictly limited to the execution evidence supplied for analysis.