Works with the AI Agents you already use
AI Observability Pro
Understand agent behavior in production with actionable traces, metrics, evaluations, SLOs, cost analysis, and incident diagnostics.
$9.99
AI Observability Pro
Example session with this skill installed
Our multi-agent support platform has rising p95 latency and cost, plus users report occasional duplicate actions. We have three agents, a model provider, vector search, and tools that update tickets. We do not yet know which stage causes the issues. Propose an end-to-end trace schema, event names, metric families, SLOs and alert rules, privacy-safe fields, dashboard panels, and a two-week instrumentation rollout. Clearly label assumptions and do not claim access to telemetry.
- Read your context and instructions
- Compiled the ai observability pro
Production Agent Observability Plan
Summary
Instrument the user-to-outcome path with correlated traces, structured transition events, a small set of actionable service and user-impact metrics, and verified action outcomes. This is a proposed design from the supplied scenario; no logs, dashboards, traces, or production systems were inspected.
Trace model
Create one trace per user request. Propagate trace_id, request_id, pseudonymous tenant_id, workflow_id, run_id, agent_run_id, and tool_call_id. Each span records span_id, parent_span_id, operation, start time, duration, status, service, agent, model and prompt version, safe attributes, error class, retry count, token usage and cost when returned by providers. Suggested spans: request.receive, auth.authorize, route.select, context.build, retrieval.query, model.generate, tool.select, approval.wait, action.execute, action.verify, output.validate, response.deliver. Never capture hidden reasoning or raw credentials.
Events and duplicate-action evidence
Emit RUN_STARTED, CONTEXT_READY, MODEL_COMPLETED, TOOL_REQUESTED, TOOL_SUCCEEDED, TOOL_FAILED, APPROVAL_REQUESTED, ACTION_SUBMITTED, ACTION_VERIFIED, FALLBACK_USED, POLICY_BLOCKED, RUN_COMPLETED, and RUN_FAILED. Include timestamp, trace_id, run_id, component/version, severity, privacy class, and a short redacted payload summary. For ticket updates, attach a stable idempotency key and external_action_id; separately record request, provider acknowledgement, and verified resulting ticket state. A timeout is an unknown outcome until reconciled.
Metrics and SLO candidates
- User-visible completion rate, request rate, workflow failure rate, model/tool error rate, retry and fallback rate.
- End-to-end and per-stage p50/p95/p99 latency, especially queue, retrieval, model, tool, approval, verification, and delivery.
- Duplicate side-effect rate and percentage of actions with verified outcomes.
- Cost per successful run, model/token and tool costs where measurable, and quality evaluation pass rate.
- Candidate SLO: at least 99% of eligible requests complete successfully within the agreed latency objective over a rolling 28-day window. Set the latency threshold and exclusions only after baseline and product agreement.
Page on sustained user-impacting failure or SLO burn, not every anomaly. Warn on latency/cost trends and growing unverified actions. Each alert needs an owner, runbook, and useful trace link. Avoid tenant IDs or request IDs as metric labels; use bounded dimensions such as agent, model version, tool, region, and outcome class.
Dashboard panels
- User-visible success and request volume by agent/version.
- End-to-end p50/p95/p99 and stacked stage latency.
- Tool request, acknowledgement, verification, timeout, retry, and duplicate-action rates.
- Cost per successful run alongside quality pass rate.
- Error and policy-block rates with links to redacted causal traces.
Two-week rollout
Week 1: agree on privacy and retention; define semantic conventions and correlation propagation; instrument request, model, retrieval, and tool spans; add action idempotency and verification events in staging. Week 2: validate with synthetic failure, timeout, retry, and duplicate scenarios; compare trace totals to provider billing and ticket state; establish baseline; review SLOs and alerts with owners; gradually release and retain rollback.
Open decisions
Confirm latency target, success definition, retention period, acceptable sampling, evaluation method, idempotency support, and incident owner. Do not conclude root cause until traces or controlled tests distinguish correlated stages.
Connects securely to your tools. The creator never sees your data.
About this skill
AI Observability Pro
Design practical observability for AI agents in production. Map each request from intake through context assembly, model and tool activity, approvals, verified external actions, and the user outcome. Connect every signal to an operational, product, reliability, safety, quality, or cost decision.
What it helps with
- Map agent topology and user journeys across services, workers, models, retrieval, memory, tools, queues, and external APIs.
- Define causal traces, structured events, logs, and stable identifiers with versioned context and safe tenant-aware correlation.
- Build metric families for traffic, success, latency, errors, quality, tooling, retrieval, memory, safety, cost, and infrastructure.
- Design actionable SLOs, alert thresholds, dashboards, incident evidence, and rollout checks.
- Investigate regressions, hidden retries, latency spikes, token spend, tool failures, policy blocks, retrieval issues, and duplicate side effects.
Operating principles
Observe execution metadata and verified outcomes, never private chain-of-thought. Instrument before guessing. Preserve causality and distinguish a tool request from confirmed completion, a model confidence score from correctness, and request success from user success. Protect secrets and personal data; use pseudonymous identifiers when sufficient. Sample carefully so rare failures remain visible. Avoid high-cardinality labels and alerts without a response action. Correlation alone does not prove root cause.
How to use it
Share the system purpose, architecture, providers, workflow, model and prompt versions, tools, retrieval/memory, tenant model, existing telemetry, data sensitivity, retention, SLOs, incidents, costs, and current pain points. Identify what evidence is available. The skill will state assumptions, propose instrumentation and metrics, and prioritize next steps without claiming production access or measured results.
Boundaries
This skill designs and diagnoses from supplied or authorized evidence. It does not invent traces, integrations, dashboards, incidents, instrumentation, root causes, or completed actions. Avoid collecting raw chain-of-thought, credentials, or unrestricted private context.
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
Findings depend on telemetry completeness, consistent identifiers, and reliable provider/tool outcomes. Sampling can hide rare events, while excessive labels can raise cost and impair performance. Thresholds and SLOs need real baselines and owner agreement. Root cause and action completion must be established from evidence, not inferred from correlation.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 month ago
Permissions
Read telemetry, configuration, or code only when the user authorizes the source and scope. Do not collect raw chain-of-thought, credentials, or unnecessary personal data. Do not claim access, instrumentation, incident findings, or successful actions without verifiable evidence.