AI Agent Product Analytics Workbench

DeskPilot completion audit — synthetic demonstration

Technical acknowledgement is 11/12 (91.67%), but only 9/12 tasks are state-confirmed (75%) and 7/12 have the intended durable outcome (58.33%). One of ten completion claims is false: task t09 was acknowledged and claimed complete, but the correct support record was not state-confirmed. That silent failure remains an explicit gate even if other metrics improve.

Supervision decision — recommendation: use supervision for this bounded support-record workflow and require an independent expected-state read before saying the work is complete. Route unverified or failed work to a visible recovery/escalation state. Do not expand autonomous use based on acknowledgement rate. A proposed product owner should own this decision before the next release review; no actual owner or calendar release date was supplied.

All data is fictional. This audit uses only packages/ai-agent-product-analytics-workbench/fixtures/source-packet.md and fixtures/agent-tasks.csv. No live account or external action is involved.

Population, window, and evidence contract

The complete supplied population is 12 eligible support-record tasks from a seven-day cohort whose seven-day durability observation window is complete. Exact calendar dates, event timestamps, version, autonomy mode, and extraction time are absent. The unit is one distinct synthetic task ID; all 12 IDs are unique. No retry rows or corrections are supplied, so each row is counted once. Results cover all supplied tasks, not production across customers or versions.

The CSV represents technical signals and independent record reads, as asserted by the source packet. Its booleans are

instrumented facts within the synthetic fixture. There are no raw tool responses or authoritative record snapshots with which to reproduce those reads. The observations identify an intentional reversal (t07), loss of intended usable state (t08), false completion (t09), acknowledgement without completion (t10), and execution failure (t12). No human/model scoring rubric is supplied; there are no evaluation-judgment scores in this audit.

The intended behavior is creation of the correct usable support record and honest communication of its status. The likely immediate beneficiary is the support operator; that is an assumption, not evidence of user satisfaction. User effort, acceptance, revenue, retention, cost, and durable net value against a counterfactual are not measured. Supervised versus autonomous execution cannot be inferred from these rows.

Metric contracts and calculated results

All population metrics use

12 eligible tasks. Claim-verification metrics use

10 tasks with a completion claim, not 12. These contracts inform the completion-language and supervision decision. The source is the full synthetic CSV with complete stated durability coverage; join/deduplication is distinct task_id, with no attempt-level history available.

An explicit attempted event is missing. Eleven acknowledged rows entail an execution attempt; t12's “execution failed” observation suggests the remaining task was attempted too. Thus 12 attempts are a reasonable

inference, but an independently instrumented attempt rate is unavailable. Do not silently turn eligible tasks into an audited attempted-task denominator.

Preserve the state chain

Attempted, acknowledged, claimed, state-confirmed, user-valued, and durable are separate states. Acknowledgement does not advance a task directly to verified completion. User-valued state is unobserved here. A later intentional reversal is a different outcome from losing the intended usable state: t07 is not automatically a product malfunction, while t08 is a durability failure described by the fixture. Both remain in the stated 7/12 unadjusted durable-outcome metric; no denominator is removed after observing an unfavorable outcome.

The most serious silent failure is t09: the user was told the work was complete despite the absence of the correct requested support record in the supplied state check. The fixture identifies the failure, but no trace identifies whether it originated in orchestration, a tool response, expected-state matching, or completion-language policy. t10 shows a similar technical-versus-state gap without a completion claim. t12 exposes an execution failure; it is not a false completion. The timing of the checks relative to claims is unavailable.

Minimum instrumentation and verification plan

This is a

recommendation, not an implemented schema or a claim of current access. The responsible engineering/analytics owner should produce an instrumented pilot and evidence review before autonomy expansion.

Every event needs pseudonymous task_id, event ID, event time and ingestion time, version, bounded intent type, and autonomy mode. Execution-related events additionally need execution_id; tool events need their call ID. Keep a task separate from a conversation, session, tool call, and affected record. Use durable identifiers; label any timestamp-only fallback. Retain only the data needed to verify this job.

Task-level metrics count a durable task once after its required outcome is checked; attempt-level metrics retain all attempts. An idempotent retry creating one correct record is not multiple outcomes. A duplicate record or later repair should remain visible as a correction/integrity issue. Compare event times and ingestion lag before assessing claim order. Set least-privilege read access, a documented retention period, and deletion propagation across task, trace, and evidence stores; exact durations require the deployment's privacy contract.

Minimum review cuts are outcome group and false-claim status. New instrumentation should add intent complexity, autonomy mode, surface/tool, version, and cohort before claiming reliability. Review contrastive traces for t09, t10, t08, and a verified durable success; use population data to assess recurrence. No trace inspection or causal experiment was possible with this fixture.

Proposed quality gates — hypotheses, not measured standards

Keep the seven material quality claims separate. The following floor/target/delight levels are design proposals requiring validation in a representative pilot; no achieved production threshold is asserted. Critical false completion cannot be offset by speed, cost, or acknowledgement success.

The zero-unverified-claim floor is a proposed integrity gate, not a statistical promise that future false claims are impossible. These twelve tasks cannot determine a production target, confidence bound, release sample size, latency budget, or cost tradeoff. A faster tool path has little value if it produces confident false completion; verification adds a read and latency whose measured cost must be considered separately.

Next action and reproduction

One next test: the engineering owner should replay the bounded support-record path with synthetic expected-state assertions, covering an acknowledged but absent record, intentional reversal, lost usable state, and correct durable success. Confirm that only a matching independent pre-claim read permits completion language, and that non-matches remain unresolved/escalated. Review the event joins and delayed-check results with the product owner before the next autonomy decision. No actual release date was supplied.

The optional helper was executed and independently checked:

python scripts/summarize_tasks.py fixtures/agent-tasks.csv

Run from the package directory with Python 3. It uses only the named CSV and the standard library. It prints the observed counts, eligible-task population rates, claim verification denominator, and explicit false-completion ID; it does not implement the proposed verification plan.