Apm Alert

Apm Alert

# APM Alert Designer

Most teams set up monitoring but fail at alerting. The result is not poor observability — it's alert fatigue: 47% of on-call engineers report that noise from poorly tuned alerts delays incident response by 15+ minutes. Alertmanager configurations grow organically into unmaintainable spaghetti. Rules fire for infrastructure symptoms (pod restarts, disk usage) while genuine user-impacting degradations go unnoticed. Spikes in pager volume trigger desensitization, and critically, no two teams define "critical" the same way, making on-call handoffs unreliable. This skill solves what monitoring tools don't: the discipline of designing, naming, testing, routing, and maintaining alert rules that are actionable, unambiguous, and actually reduce MTTR.

What It Does

• Generates Prometheus alerting rules from YAML/JSON specs with consistent naming, labels, and severity
• Produces Alertmanager route trees with team-based notification routing (email, Slack, PagerDuty, Opsgenie, webhook)
• Validates PromQL expressions for syntax correctness and common anti-patterns
• Detects alert drift between declared rules and live Prometheus/Alertmanager instances
• Creates Grafana-managed alert rule definitions for provisioning
• Generates runbook templates linked to each alert by severity and team

Frameworks and Standards Covered

| Standard/Framework | Application | |---|---| | Prometheus Alerting Rules (PromQL) | Rule generation, validation, and live comparison | | Alertmanager Configuration (v2) | Route trees, receivers, grouping, inhibition, silencing | | Grafana Unified Alerting (v10+) | Grafana-managed rule provisioning | | Google SRE Workbook | Symptom-based vs cause-based alerting, burn-rate alerts | | ITIL Incident Management | Alert severity classification, escalation paths | | OpenTelemetry | Metric-to-alert mapping across OTel data sources | | PagerDuty/Opsgenie API | Notification routing and on-call integration | | Datadog Monitor API | Cross-platform alert rule generation |

Detailed Feature Breakdown

### 1. Alert Rule Generation Accepts a YAML or JSON specification of the desired alerts and produces ready-to-deploy Prometheus recording and alerting rules. Each rule gets: - Consistent naming with team prefix and severity suffix - Proper `for` duration to prevent flapping - Label annotations (summary, description, runbook, dashboard link) - Severity mapping (critical/warning/info) with distinct notification paths ```yaml # Example input spec alerts: - name: high_error_rate expr: 'rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01' severity: critical for: 5m team: platform-sre runbook: docs/runbooks/high-error-rate.md dashboard: https://grafana.example.com/d/abc123 ``` ### 2. Alertmanager Route Configuration Generates an `alertmanager.yml` route tree from a team/routing spec: ```yaml routes: - team: platform-sre matchers: - severity = critical receiver: pagerduty-critical grouping: group_wait: 30s group_interval: 5m repeat_interval: 4h - team: platform-sre matchers: - severity = warning receiver: slack-alerts ``` ### 3. PromQL Validation Validates generated PromQL expressions for: - Syntax correctness (parseable by Prometheus parser) - Common anti-patterns (missing `rate()`, wrong vector matching, histogram misuse) - Missing metric name prefix conventions - Rate interval longer than scrape interval ### 4. Drift Detection Compares declared alert rules (from YAML spec) against live Prometheus `/api/v1/rules` and Alertmanager `/api/v2/status` endpoints. Reports: - Missing rules - Threshold drifts (spec says `> 0.05`, live says `> 0.1`) - Severity mismatches - Orphaned rules (in live but not in spec) ### 5. Grafana Provisioning Generates Grafana-managed alert rule definitions in the provisioning format: ```json { "apiVersion": 1, "groups": [{ "orgId": 1, "name": "platform-sre-alerts", "folder": "Platform SRE", "rules": [...] }] } ``` ### 6. Runbook Generation Each alert gets a generated runbook template with pre-filled sections: - What triggers this alert - Severity and escalation path - Query to investigate - Common causes - Remediation steps (ordered by blast radius) - Post-mortem links ### 7. Scenario Simulation Simulates alert firing conditions by running queries against time-shifted metric data. Validates: - Alert fires when expected - `for` duration is appropriate - Resolves when condition clears - Notification routes to correct receivers

Usage

### Quick Start ```bash # 1. Define your alert spec apm-alert spec init --project my-app # 2. Generate Prometheus rules apm-alert generate rules --spec alerts.yaml --output rules/ # 3. Generate Alertmanager config apm-alert generate routes --spec teams.yaml --output alertmanager.yml # 4. Validate generated rules apm-alert validate --rules rules/*.yml # 5. Detect drift from live Prometheus apm-alert drift --prometheus http://localhost:9090 --spec alerts.yaml ``` ### Configuration Example `apm-alert.yaml`: ```yaml project: my-app environment: production default_runbook_base: docs/runbooks/ default_dashboard_base: https://grafana.example.com/d/ default_team: platform-sre prometheus_url: http://localhost:9090 alertmanager_url: http://localhost:9093 notification_receivers: pagerduty: routing_key: ${PAGERDUTY_KEY} slack: webhook_url: ${SLACK_WEBHOOK} ```

Output Format

### Generated Rule File (`rules/production-alerts.yml`) ```yaml groups: - name: production-critical interval: 30s rules: - alert: HighErrorRate expr: rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m]) > 0.01 for: 5m labels: severity: critical team: platform-sre annotations: summary: "High HTTP error rate on {{ $labels.instance }}" description: "Error rate is {{ $value | humanizePercentage }} for the last 5 minutes" runbook: docs/runbooks/high-error-rate.md dashboard: https://grafana.example.com/d/abc123 ``` ### Drift Report ```json { "drift_detected": true, "missing_rules": ["HighLatencyP99"], "threshold_drifts": [ { "rule": "HighCPU", "spec_threshold": "> 0.8", "live_threshold": "> 0.9" } ], "orphaned_rules": ["OldDiskAlert"], "severity_mismatches": [ { "rule": "MemoryPressure", "spec_severity": "warning", "live_severity": "critical" } ] } ```

Why This Beats Prompting It Yourself

Raw Prompting APM Alert Designer --- --- --- Consistency Ad-hoc naming, inconsistent labels Naming convention enforced on every rule Validation No PromQL syntax checking Full syntax + anti-pattern detection Drift None — manually track changes Automated comparison against live instances Routing Guesswork per team Declarative route tree generation Runbooks Separate manual process Auto-generated per alert with remediation Multi-env Fragile copy-paste Templated env-specific thresholds Compliance No audit trail Rule lineage and change history

Use Cases

• Platform SRE onboarding: A new SRE joins and needs to understand which alerts page whom, why, and what to do. The generated runbook collection and route map provide instant onboarding documentation.
• Production incident post-mortem: After a P0 outage, post-mortem reveals the alert existed but was routed to the wrong team. The drift detection and route validation prevent recurrence.
• Multi-team alert standardization: Ten engineering teams each have their own Prometheus rules files with no naming convention. Standardize all teams on a single consistent rule set with team-specific severity thresholds.
• SLO burn-rate alerting: Implement Google SRE Workbook burn-rate alerts (5m, 30m, 6h windows) with consistent configuration across all services in the portfolio.
• Cloud migration alert migration: Moving from Datadog to Grafana Mimir. Generate equivalent Grafana-managed alert rules from existing Datadog monitor definitions during the migration window.