=== LEGACY BANKING INCIDENT RUNBOOK === Runbook title: Overnight Core Banking Batch Job Failure Runbook System/capability: Core banking batch processing Purpose: Provide structured triage for recurring overnight batch failures that may affect account updates, downstream reconciliation, reporting, and start-of-day operations. Scope: Included: - batch failure classification - predecessor and successor job review - input/output file checks - log review checklist - escalation path - rerun decision support - downstream impact checks - post-incident summary Out of scope: - unauthorized reruns - direct database edits - ledger corrections - production changes without approval - final decision to rerun or rollback Severity model: SEV-1: Batch failure affects core ledger, account balances, settlement, or regulatory deadline. SEV-2: Batch failure threatens start-of-day operations or downstream reconciliation deadline. SEV-3: Batch failure is isolated and can be corrected within SLA. Incident triggers: - job fails with non-zero return code - job exceeds runtime threshold - predecessor job missing - input file missing - output control totals mismatch - downstream reconciliation job blocked - repeated failure across multiple nights Initial triage: 1. Confirm job name, job ID, and environment. 2. Confirm first failure timestamp. 3. Check predecessor jobs. 4. Check input file arrival. 5. Check control totals if applicable. 6. Check whether downstream settlement/reconciliation/reporting deadlines are at risk. 7. Check recent changes to batch schedule, job parameters, database, files, or vendor feed. 8. Preserve logs before retry or rerun. Dependency map: Primary job: [Batch job name] Upstream dependencies: - source transaction files - prior posting jobs - vendor file transfer - scheduler calendar Downstream dependencies: - reconciliation job - general ledger interface - regulatory reporting feed - customer statement generation - start-of-day availability Evidence to collect: - job ID - job start/end time - return code - scheduler log - application log - file arrival timestamp - input file size/control total - rejected records summary with sensitive data masked - downstream blocked jobs - recent change record - prior incident reference Log review checklist: Application/batch logs: - first error - repeated error pattern - rejected record count - file format errors - timeout errors - dependency unavailable Scheduler logs: - predecessor completion - job calendar - retry attempts - resource constraints Database metrics: - locks - connection pool - storage - deadlocks - long-running queries Decision tree: IF input file is missing: Check file transfer status. Check vendor delivery confirmation. Escalate to file transfer/vendor owner if not delivered. IF input file arrived but job rejected records: Review rejection summary. Mask sensitive data. Identify common rejection pattern. Escalate to application owner or source-system owner. IF predecessor job failed: Follow predecessor job runbook. Do not rerun current job until dependency is resolved. IF downstream deadline is at risk: Escalate to major incident process and business owner. IF rerun is considered: Confirm approval, data integrity impact, duplicate-processing risk, and downstream communication. Escalation path: Incident commander: [Major incident lead if SEV-1/SEV-2] Batch operations: [Team/contact] Application owner: [Team/contact] Database owner: [Team/contact] File transfer/vendor owner: [Team/contact] Business owner: [Operations/reconciliation owner] Compliance/regulatory contact: [If deadline or reporting risk exists] Rollback/rerun decision guidance: Rollback or rerun must be approved by authorized operations and application owners. Before rerun, confirm duplicate-processing risk, input file integrity, downstream status, and reconciliation impact. Workaround assessment: Manual processing may require dual control, audit record, business approval, and reconciliation plan. Data integrity checks: - duplicate transactions? - missing transactions? - partial processing? - control totals match? - ledger affected? - downstream reconciliation affected? Communication templates: Technical update: Batch job [name] failed at [time] with [return code]. Initial impact assessment: [impact]. Evidence collected: [evidence]. Current owner: [owner]. Next update: [time]. Business update: Overnight batch processing for [capability] is delayed. Current known impact: [impact]. Reconciliation/settlement deadline risk: [risk]. Next update: [time]. Post-resolution checks: - job completed successfully - downstream jobs resumed - control totals verified - reconciliation checks passed - business owner notified - incident record updated Post-incident summary template: Incident ID: Job: Timeline: Root cause: Impact: Resolution: Approvals: Data integrity checks: Action items: Runbook updates: Owner and review cadence: Batch operations owner. Review quarterly or after any SEV-1/SEV-2 batch incident.