Works with the AI Agents you already use
Agent Observability: Traces, Costs and SLOs for LLM Agents
If you cannot answer "what did the agent see, decide and do on run X, and what did it cost?" within five minutes, the agent is not production-ready.
$12
Agent Observability: Traces, Costs and SLOs for LLM Agents
Example session with this skill installed
We run three LLM agents in production on GCP and have no idea what they cost per task or why some runs take 5 minutes. Design the observability for them: what to trace, what to log, which metrics and alerts, and the BigQuery queries for a dashboard. Write it to observability.md.
- Read your context and instructions
- Compiled the agent observability
observability.md: one trace per run (invoke_agent → chat and execute_tool spans with gen_ai.* attributes, served model, request id, cost); event log to BigQuery via a log sink with agent_version, hashed tenant and run_finished always emitted; metrics and alerts (success rate drop vs 7-day baseline, cost per successful run +30%, budget exhaustion > 2%, tool error rate > 5% over 15 min, cache hit ratio drop > 20 points after a deploy); content capture off by default, redacted with a 30-day retention where enabled; dashboard SQL for health, cost per successful run, cache hit ratio by version, tool reliability and the slowest runs.
Connects securely to your tools. The creator never sees your data.
About this skill
If you cannot answer "what did the agent see, decide and do on run X, and what did it cost?" within five minutes, the agent is not production-ready. This skill gives your agent the telemetry to answer it.
What your agent builds with it
- One trace per run with spans for every model and tool call, using the OpenTelemetry GenAI semantic conventions; served model (not requested), provider request ids, policy decisions, cost; content capture treated as a privacy decision.
- A structured event log (JSON Schema included) for cross-run questions, with an agent_version hash so every metric splits by release, hashed tenants and a guaranteed
run_finishedrow. - Metrics, SLOs and alerts that matter: success rate, cost per successful run, budget-exhausted rate, refusal and fallback rate, tool error and timeout rates, policy denials, cache hit ratio, latency, provider errors.
- A dashboard built from six ready BigQuery queries (health, cost per successful run with time-valid prices, cache effectiveness by version, tool reliability, runaway runs, served-model mix).
- A cost report (
cost_report.py) from call logs: cost per run and per successful run, cache hit ratio, p50/p95 latency, unpriced models, grouped by agent, model or day. - Closing the loop: weekly transcript review and post-deploy comparisons by version.
Included: SKILL.md, event schema, example pricing file, BigQuery analytics SQL, cost_report.py (standard library), and an eval case.
Part of the Production AI Agents series. Pair with Agent Evals (quality) and Agent Reliability (the loop that emits the events).
Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for LLM Ops.
The example pricing file is illustrative and dated; always use your provider's current prices. It does not score answer quality (see Agent Evals) and does not include a hosted dashboard, only the queries.
How to use
- 1
Connect your agent to the Agensi MCP
Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup
- 2
Sign in to Agensi
Your agent shows a sign-in link. Sign in and click Allow.
- 3
Ask for what you need
Your agent loads this skill and uses it on your task. It always asks before buying a skill.
Reviews
No reviews yet
Be one of the first to try it. Every listed skill goes through the trust checks below.
Security scanned
Checked by our 8-point scan before listing
Fresh listing
Recently published to Agensi
Money-back guarantee
Doesn't do what its page promises? You get your money back.
Trust & safety
Security scan
Security scan passed 1 day ago
- Software
Python 3.10+
- Optional
- OpenTelemetry SDKBigQuery (or another warehouse)
Permissions
File Scopes
cost_report.py reads local JSONL and pricing files and makes no network calls.