Works with the AI Agents you already use

    Claude CodeClaude CodeCursorCursorCodex CLICodex CLIDeepSeekDeepSeekMuseMuse+22 more

    Production Incident Post-Mortem & RCA Generator

    1

    The Production Incident Post-Mortem & RCA Generator is a specialized DevOps skill designed to bring order to the chaos of live outages.

    $5

    Secure checkout via Stripe

    Money-back guarantee

    Converts to your local currency at checkout

    Security scanned
    Production Incident Post-Mortem & RCA Generator

    Production Incident Post-Mortem & RCA Generator

    Example session with this skill installed

    Generate an RCA for this incident based on the Slack transcript below:

    [14:02 UTC] @dave_ops: hey the payment gateway is throwing 500s. Pagerduty just went off.
    [14:04 UTC] @sarah_backend: looking into it. Datadog shows the checkout-service CPU is at 100%.
    [14:07 UTC] @dave_ops: Did someone deploy something? Customers are complaining on Twitter. John Doe (john.doe@gmail.com) said his card was charged twice.
    [14:12 UTC] @sarah_backend: wth. Dave, did you push that untested PR to main? The one with the new schema validator?
    [14:15 UTC] @dave_ops: Yeah I pushed commit 8f92a1b, but it worked in staging!
    [14:18 UTC] @sarah_backend: staging doesn't have the new payment provider API keys! It's looping and crashing. I see the DB connection string postgres://admin:SuperSecretProdPass!@10.0.1.45/prod_db exposed in the error logs!
    [14:22 UTC] @dave_ops: my bad, im an idiot. im rolling back the commit now.
    [14:28 UTC] @sarah_backend: ok, rollback complete. CPU is dropping. 500 errors are back to 0.
    [14:35 UTC] @dave_ops: we need to lock down the main branch so i cant do this again.

    • Read your context and instructions
    • Compiled the production incident post-mortem

    PRODUCTION INCIDENT POST-MORTEM: Payment Gateway / Checkout Service Outage

    Executive Overview

    Date of Incident: [N/A - Requires Manual Input, date not specified in provided logs]
    Incident Severity Level: High
    Total Downtime / Degradation: 26 minutes (14:02 UTC – 14:28 UTC)
    Mean Time to Detect (MTTD): [N/A - Requires Manual Input, deployment timestamp not provided]
    Mean Time to Resolve (MTTR): 26 minutes

    1. Executive Summary

    The checkout service experienced a full CPU saturation event causing widespread HTTP 500 errors on the payment gateway, impacting customer checkouts and resulting in at least one reported duplicate-charge incident. The structural root cause was a lack of environment parity between staging and production for payment provider credentials, combined with the absence of branch protection and pre-merge validation gates on the main branch. The incident was resolved by rolling back the triggering commit, after which service CPU and error rates returned to baseline.

    2. Root Cause Analysis (RCA) & The 5 Whys

    A new schema validator was merged directly to the main branch and deployed to production. The validator functioned correctly in staging, but staging does not carry the production payment provider API keys. In production, the missing credentials caused the validator/payment integration logic to enter a repeated retry-and-crash loop, driving checkout-service CPU utilization to 100% and producing cascading HTTP 500 responses on the payment gateway. During triage, a production database connection string was also found exposed in plaintext error logs, indicating a secondary secrets-handling gap surfaced by the incident.

    The 5 Whys Chain:

    Why did the incident occur? Because a newly deployed schema validator caused the checkout service to enter a crash loop, driving CPU to 100% and triggering HTTP 500 errors on the payment gateway.
    2.

    Why? Because the service lacked the required production payment provider API keys, causing the validator's calls to the payment integration to fail and retry indefinitely.
    3.

    Why? Because the staging environment used for pre-deployment validation did not have configuration parity with production (missing payment provider credentials).
    4.

    Why? Because there was no automated environment-parity check or integration test in the deployment pipeline that would have caught a staging/production configuration mismatch prior to merge.
    5.

    Why? Because the main branch lacked mandatory branch protection, code review, and pre-merge CI gating, allowing an untested change to be pushed directly to production-facing code.

    3. Integrated Incident Timeline

    Note: only timestamps present in the source transcript are shown. The exact deployment time of the triggering commit was not stated and is marked accordingly.

    Timestamp (UTC)Source / ActorEvent / Action Taken
    [N/A - Requires Manual Input]DeployerSchema validator commit (8f92a1b) deployed to production main branch (exact deployment time not provided in source data).
    14:02:00PagerDuty / On-Call EngineerAlert fired for HTTP 500 errors on the payment gateway. Incident triage began.
    14:04:00On-Call Engineer (Backend)Triage started. Datadog telemetry reviewed; checkout-service CPU utilization observed at 100%.
    14:07:00On-Call Engineer (Operations)Customer impact reports received via external social channels, including a reported duplicate transaction charge.
    14:12:00–14:18:00On-Call EngineersRoot cause identified: recently merged commit lacked production payment provider credentials, causing a crash loop. A production database connection string was also observed exposed in plaintext error logs.
    14:22:00On-Call Engineer (Deployer)First mitigation attempt: rollback of commit 8f92a1b initiated.
    14:28:00On-Call Engineer (Backend)Resolution: Rollback completed. CPU utilization and HTTP 500 error rate returned to baseline (0).

    4. System & Business Impact

    Blast Radius: Checkout service and payment gateway; at least one customer-reported duplicate transaction charge. Full percentage of affected sessions is [N/A - Requires Manual Input].
    *

    Technical Metrics: Checkout-service CPU utilization peaked at 100%; HTTP 500 error volume on the payment gateway [N/A - Requires Manual Input, exact error count not provided].

    • SLA Status: [N/A - Requires Manual Input]

    5. SMART Corrective Actions

    • [ ]

    [Immediate - 48 Hours] Enable branch protection rules on the main branch requiring at least one peer review and a passing CI pipeline before merge.

    • Owner: DevOps / Platform Team
    • [ ]

    [Immediate - 48 Hours] Audit application and error-logging configuration to ensure database connection strings and credentials are never written to logs in plaintext; rotate the exposed production database credential immediately.

    • Owner: Backend Team
    • [ ]

    [Short-Term - 1 Week] Implement an automated staging/production configuration-parity check in the CI/CD pipeline that validates required environment variables and API credentials exist in all target environments prior to deployment approval.

    • Owner: Backend Team
    • [ ]

    [Short-Term - 1 Week] Configure automated Datadog alerting on checkout-service CPU utilization exceeding 80% sustained for more than 2 minutes, in addition to existing 500-error alerting.

    • Owner: Backend Team
    • [ ]

    [Long-Term - 4 Weeks] Introduce a canary or staged rollout deployment strategy for the checkout/payment integration path, with automated rollback triggered on error-rate or CPU threshold breaches.

    • Owner: Architecture Guild
    • [ ]

    [Long-Term - 4 Weeks] Implement a secrets-management solution (e.g., vault-based credential injection) to eliminate the practice of embedding raw connection strings in application configuration reachable by logs.

    • Owner: Platform Team

    🛠️ PLATFORM INTEGRATION PAYLOADS

    A. Jira Cloud / Server Format (Wiki Markup Syntax)

    Copy and paste the block below directly into the description or comment box of your Jira RCA/Problem ticket.

    h1. PRODUCTION INCIDENT POST-MORTEM: Payment Gateway / Checkout Service Outage
    
    ||Date||[N/A - Requires Manual Input]||Severity||High||MTTR||26 mins||
    
    
    h3. Executive Summary
    {quote}The checkout service experienced full CPU saturation causing HTTP 500 errors on the payment gateway, including a reported duplicate customer charge. The root cause was a staging/production configuration parity gap for payment provider credentials, compounded by a lack of branch protection and pre-merge validation. The issue was resolved via rollback of the triggering commit.{quote}
    
    h3. Root Cause Analysis (5 Whys)
    
    # *Trigger:* A newly deployed schema validator caused the checkout service to enter a crash loop, driving CPU to 100% and triggering HTTP 500 errors.
    # *System State:* The service lacked required production payment provider API keys, causing repeated failed retries.
    # *Process Gap:* Staging environment lacked configuration parity with production for payment provider credentials.
    # *Pipeline Gap:* No automated environment-parity or integration test existed in the deployment pipeline to catch this mismatch pre-merge.
    # *Architectural Root:* Main branch lacked mandatory branch protection, code review, and CI gating.
    
    h3. SMART Corrective Actions (Create linked Jira tasks for these)
    * *[Immediate]* Enable branch protection with mandatory review and passing CI on main (Component: DevOps / Platform Team)
    * *[Immediate]* Rotate exposed production DB credential and audit logging for plaintext secrets (Component: Backend Team)
    * *[Short-Term]* Add automated staging/production config-parity check to CI/CD pipeline (Component: Backend Team)
    * *[Short-Term]* Add CPU-utilization alerting for checkout-service (Component: Backend Team)
    * *[Long-Term]* Implement canary rollout with automated rollback for payment integration path (Component: Architecture Guild)
    * *[Long-Term]* Adopt secrets-management/vault solution for credential injection (Component: Platform Team)
    

    B. ServiceNow Field-Mapping Payload

    Map the values generated below into the standard columns of the ServiceNow incident or problem record schema.

    Short Description (short_description): Outage RCA: Checkout Service / Payment Gateway - HTTP 500 errors from CPU saturation - [N/A - Requires Manual Input]

    • Description (description):
    === SYSTEM INCIDENT SUMMARY ===
    Downtime: 26 Minutes
    Blast Radius: Checkout service, payment gateway; at least one duplicate-charge customer report
    
    === PRIMARY FAIL-VECTOR ===
    A schema validator commit (8f92a1b) was deployed to production without required payment provider API credentials present in that environment. This caused the checkout service's payment integration calls to fail and retry continuously, saturating CPU at 100% and producing cascading HTTP 500 responses on the payment gateway. A production database connection string was also found exposed in plaintext error logs during triage. [REDACTED] indicates sanitized credential/PII data removed from source logs.
    
    === RECONSTRUCTED HIGH-LEVEL TIMELINE ===
    - Detection: 14:02:00 UTC - PagerDuty alert triggered for payment gateway HTTP 500 errors.
    - Mitigation: 14:22:00 UTC - Rollback of commit 8f92a1b initiated.
    - Resolution: 14:28:00 UTC - Systems validated; CPU and error rates returned to baseline.
    

    Resolution Codes / Notes (close_notes): System restored to operational baseline via rollback of commit 8f92a1b to the prior production-stable state. CPU utilization and payment gateway error rate confirmed at baseline post-rollback.
    *

    Priority / Impact / Urgency Fields: Impact = 2-Significant/Multiple Users; Urgency = 1-High -> Priority = 2-High. Exact organizational impact percentage is [N/A - Requires Manual Input].


    Data Sanitization Note: All customer PII (name, email address) and the exposed production database connection string/credential present in the raw source transcript have been redacted from this report in accordance with data privacy guardrails. Raw log retention and credential rotation should be handled per your organization's incident-response and secrets-management policy.

    Connects securely to your tools. The creator never sees your data.

    About this skill

    The Production Incident Post-Mortem & RCA Generator is a specialized DevOps skill designed to bring order to the chaos of live outages. It ingests unstructured triage data—such as Slack/Teams chat transcripts, PagerDuty alerts, error logs, and Git commit hashes—and transforms them into a structured, objective post-mortem report.

    Emphasizing a blameless "5 Whys" methodology, this skill bypasses surface-level human error to isolate the true systemic vulnerabilities in your architecture or CI/CD pipeline. Version 1.2.0 introduces strict anti-hallucination guardrails and automated data sanitization, ensuring that passwords, API keys, and customer PII are instantly redacted before the report is generated. Designed to accelerate resolution workflows, it categorizes fixes into SMART corrective action items and outputs ready-to-paste payloads perfectly formatted for Jira Wiki Markup and ServiceNow.

    Find more skills like this on the Claude Skills Marketplace, the Codex Skills Marketplace and the Gemini Skills Marketplace, or browse Claude Skills for DevOps & Cloud.

    Requires raw logs or chat data with timestamps to generate a timeline. Cannot access live environments directly; relies on user-provided text inputs.

    How to use

    1. 1

      Connect your agent to the Agensi MCP

      Copy the prompt and paste it into Claude, ChatGPT, Cursor or any agent that supports MCP. It sets up the connection for you. Manual setup

    2. 2

      Sign in to Agensi

      Your agent shows a sign-in link. Sign in and click Allow.

    3. 3

      Ask for what you need

      Your agent loads this skill and uses it on your task. It always asks before buying a skill.

    Reviews

    No reviews yet

    Be one of the first to try it. Every listed skill goes through the trust checks below.

    Security scanned

    Checked by our 8-point scan before listing

    Fresh listing

    Recently published to Agensi

    Money-back guarantee

    Doesn't do what its page promises? You get your money back.

    Trust & safety

    Security scan

    Security scan passed 2 months ago

    Permissions

    No special permissions declared or detected

    Listed2 months ago

    What's inside

    Frequently Asked Questions

    This skill ingests fragmented data from chat logs, PagerDuty alerts, and telemetry to automatically generate a structured, blameless post-mortem report, saving SRE teams hours of manual documentation while ensuring systemic root causes are identified.

    Yes, the skill includes automated sanitization guardrails that identify and redact sensitive information like API keys, passwords, and PII from your logs before the final report is generated.

    The skill provides pre-formatted markdown and data payloads specifically optimized for easy copy-pasting or API-delivery into Jira tickets, Jira Wiki (Confluence), and ServiceNow incident records.

    You receive the production-ready AI agent skill, a comprehensive integration guide for connecting your log sources, and pre-configured templates for '5 Whys' and SMART action item generation.

    While designed for high-performance LLMs, this skill is compatible with most enterprise-grade AI agents that support text and file ingestion, including ChatGPT Enterprise, Claude for Business, and custom internal DevOps assistants.

    Updates are provided periodically to enhance data sanitization patterns and expand formatting options; buyers will have access to all version 1.x improvements and bug fixes via their marketplace dashboard.

    New in DevOps & Cloud