Observability SLO Architect

You don't have a reliability problem, you have a measurement problem. This skill turns "is the service up?" into quantifiable SLOs, error budgets, and alerts that wake the right people at the right time.

What it does

End-to-end SLO engineering:

• SLI selection — 12 indicators across availability, latency, throughput, durability, correctness
• SLO framework — pick the right SLO for your service (8 patterns)
• Error budget calculation — convert SLO to downtime budget per window
• Burn-rate alerts — fast burn (1h, 6h) vs slow burn (24h, 3d)
• Prometheus rules — ready-to-use recording + alerting rules
• Grafana dashboards — SLO compliance, burn rate, error budget remaining
• PagerDuty / Opsgenie policies — escalation + paging based on burn rate
• Runbook templates — what to do when SLO is at risk
• Review process — monthly SLO review template

When to use it

• You're launching a new service and need to define SLOs
• Your on-call is exhausted from alerts that don't matter
• Customers complain about reliability but you can't quantify it
• You want to introduce error budgets but don't know how
• Your SLOs are aspirational ("99.9%") but no one's held to them
• You need to migrate from "uptime" to SLO-based thinking

Why it's better than ad-hoc prompting

Most "define SLOs" prompts give textbook answers. This skill is different:

• 8 specific SLO frameworks — pick the right one for your service type
• Ready-to-deploy rules — Prometheus + Grafana, not pseudocode
• Burn-rate alerts done right — Google's 2% / 50% burn model
• Concrete tradeoffs — "user-facing latency" vs "internal availability"
• Templates — copy-paste runbooks, review docs

Architecture

┌─────────────────────────────────────────────────────────┐
│               Agent (Claude/Cursor)                     │
│  - Reads service description                            │
│  - Asks about user journey, criticality                 │
│  - Generates SLO spec + rules + dashboards              │
└───────────────┬─────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────────┐
│        skills/observability-slo-architect/             │
│  scripts/                                                │
│    ├── design_slo.py       # SLI selection + SLO targets │
│    ├── error_budget.py     # Budget calculation          │
│    ├── gen_prometheus.py   # Recording + alerting rules  │
│    ├── gen_grafana.py      # Dashboard JSON              │
│    ├── gen_pagerduty.py    # Escalation policy           │
│    ├── gen_runbook.py      # Markdown runbook            │
│    └── review.py           # Monthly SLO review template │
│  references/                                             │
│    ├── 8-frameworks.md     # Which SLO pattern when      │
│    ├── sli-catalog.md      # 12 SLI definitions          │
│    ├── burn-rate-guide.md  # Fast vs slow burn           │
│    └── google-sre-book-notes.md                          │
│  templates/                                              │
│    ├── slo-spec.yaml                                   │
│    ├── prometheus-rules.yaml                           │
│    ├── grafana-dashboard.json                          │
│    ├── pagerduty-policy.yaml                           │
│    └── runbook.md.tmpl                                 │
└─────────────────────────────────────────────────────────┘

Quick start

# 1. Install
pip install pyyaml

# 2. Design SLOs for a service
python scripts/design_slo.py --service "checkout-api" --user-facing --criticality high --out slo.yaml

# 3. Calculate error budget
python scripts/error_budget.py --slo 99.9 --window 30d

# 4. Generate Prometheus rules
python scripts/gen_prometheus.py --slo slo.yaml --out prometheus-rules.yaml

# 5. Generate Grafana dashboard
python scripts/gen_grafana.py --slo slo.yaml --out dashboard.json

# 6. Generate PagerDuty escalation
python scripts/gen_pagerduty.py --slo slo.yaml --out pagerduty.yaml

# 7. Generate runbook
python scripts/gen_runbook.py --slo slo.yaml --service "checkout-api" --out runbook.md

# 8. Monthly review template
python scripts/review.py --slo slo.yaml --month 2026-06 --out review-2026-06.md

The 8 SLO frameworks (which to use)

| # | Service type | Primary SLO | Secondary SLO | |---|--------------|-------------|---------------| | 1 | User-facing API | Availability (99.9%) | Latency p99 (300ms) | | 2 | Read-heavy DB | Query success rate | Read latency p95 | | 3 | Write-heavy DB | Durability (zero loss) | Write latency p99 | | 4 | Async worker | Throughput (events/min) | End-to-end latency p99 | | 5 | Batch job | Job success rate | Duration p95 | | 6 | CDN | Cache hit rate | Edge latency p95 | | 7 | Auth service | Auth success rate | Token issuance latency | | 8 | Internal API | Request success rate | Internal latency p95 |

The 12 SLIs (catalog)

Availability (3)

• http_requests_total{status!~"5.."} / http_requests_total
• uptime_seconds / total_seconds
• successful_requests / total_requests

Latency (4)

• histogram_quantile(0.99, ...) (p99)
• histogram_quantile(0.95, ...) (p95)
• histogram_quantile(0.50, ...) (p50)
• requests under SLO / total requests (good / total)

Throughput (2)

• events_processed_total / time_window
• bytes_processed_total / time_window

Durability (2)

• data_loss_bytes / total_bytes_stored
• successful_writes / total_write_attempts

Correctness (1)

• correct_responses / total_responses (where "correct" is defined per service)

Burn-rate alerts (Google SRE model)

For a 30-day SLO with 99.9% target (43.2 min budget):

| Alert | Window | Burn rate | Severity | Page? | |-------|--------|-----------|----------|-------| | Fast burn (1h) | 1h | 14.4x | Page on-call | Yes | | Fast burn (6h) | 6h | 6x | Page on-call | Yes | | Slow burn (24h) | 24h | 3x | Slack | No | | Slow burn (3d) | 3d | 1x | Slack | No |

The math: at 14.4x burn for 1h, you'd consume 1% of the 30-day budget. 14.4 * 1h / 720h = 2%. So this alert fires when 2% of budget is consumed in 1h.

Sample output (Prometheus rules)

groups:
- name: checkout-api-slo
  interval: 30s
  rules:
  # Recording: success ratio
  - record: slo:service_success_ratio:5m
    expr: |
      sum(rate(http_requests_total{service="checkout-api",status!~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="checkout-api"}[5m]))

  # Recording: error budget remaining
  - record: slo:error_budget_remaining:30d
    expr: |
      1 - (
        (1 - avg_over_time(slo:service_success_ratio:5m[30d]))
        / (1 - 0.999)
      )

  # Fast burn (1h, 14.4x)
  - alert: SLO_FastBurn_1h
    expr: |
      (1 - slo:service_success_ratio:5m)
      > (1 - 0.999) * 14.4
    for: 2m
    labels:
      severity: critical
      slo: checkout-api
    annotations:
      summary: "checkout-api SLO burning fast (1h)"
      runbook: "https://wiki/runbooks/checkout-api-slo"

  # Slow burn (24h, 3x)
  - alert: SLO_SlowBurn_24h
    expr: |
      (1 - avg_over_time(slo:service_success_ratio:5m[24h]))
      > (1 - 0.999) * 3
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "checkout-api SLO burning slow (24h)"

Pricing

Single-purchase, lifetime access. $9.00.

Includes:

• 7 Python scripts (design, budget, prometheus, grafana, pagerduty, runbook, review)
• 4 reference docs (8 frameworks, 12 SLIs, burn rate, SRE book notes)
• 5 templates (SLO spec, Prometheus rules, Grafana dashboard, PagerDuty, runbook)
• Future updates for the same major version

Example usage

"We just launched a new checkout API. Define SLOs, generate Prometheus rules, and a Grafana dashboard for the team."

The skill will:

1. Ask 5-6 questions (user-facing? criticality? tier?)
2. Recommend SLOs (e.g. 99.9% availability, p99 < 300ms)
3. Calculate error budget (43.2 min/month)
4. Generate Prometheus rules (recording + burn-rate alerts)
5. Generate Grafana dashboard JSON
6. Generate runbook
7. Output slo-package/ ready to commit

Compatibility

Works with any agent that supports the SKILL.md standard and can execute Python: Claude Code, OpenClaw, Codex CLI, Cursor, Gemini CLI, Cline, Windsurf, Aider. Tested on Linux, macOS, Windows. Outputs standard Prometheus + Grafana formats; PagerDuty YAML imports directly.

Tags

observability, slo, sli, sre, prometheus, grafana, monitoring, devops, reliability