Asynchronous Operation and Job Contract Design

Wrote docs/architecture/tasks/banking-async-001/async-design/async-operation-contract.md. Complete asynchronous operation specification establishing HTTP 202 Accepted polling contracts, Retry-After pacing, job lifecycle state machines, and RFC 9457 error details.

skill: async-design

Async Operation & Job Contract: Tax Statement Generation [ASYNC-TAX-001]

Summary

This specification establishes the asynchronous HTTP operation contract, job polling protocol, lifecycle state machine, and webhook completion architecture for tax-statement-job-service v2.0 under run ID banking-async-001. It governs long-running cryptographic PDF statement generation spanning 6.5 million wealth accounts with execution durations ranging from 45 seconds to 8 minutes. It decisively eliminates the connection pool exhaustion and thread starvation demonstrated in incident INC-4913 (where attempting to hold open 10-minute synchronous HTTP connections crashed edge API gateways, taking down web banking for 50 minutes). The contract enforces

RFC 7231 HTTP 202 Accepted job initiation, canonical status polling endpoints via the Location header, dynamic exponential

Retry-After pacing (preventing polling storms), a deterministic five-state lifecycle machine, signed webhook callbacks, and 24-hour immutable result artifact retention.

Detailed Description

Holding synchronous HTTP request-reply sockets open for operations exceeding 5–10 seconds is an anti-pattern in distributed cloud architectures. Cloud load balancers, reverse proxies, and browser clients enforce rigid connection timeouts (typically 30–60 seconds). When long-running tasks block server worker threads, sudden bursts of inbound requests rapidly exhaust connection pools. Asynchronous job protocols decouple the request submission from execution, returning an immediate status handle that clients query without holding server threads captive.

Client Ingress: POST /v1/tax-statements/jobs (450 req/sec)
                       │
                       ▼ (Job Accepted in < 15 ms)
[ HTTP 202 Accepted Response ]
  ├── Status: `HTTP/1.1 202 Accepted`
  ├── Location: `https://api.bank.internal/v1/tax-statements/jobs/job_991204`
  └── Retry-After: `30` (Directs Client to Sleep 30s Before First Poll)
                       │
                       ▼ (Client Polls GET /v1/tax-statements/jobs/job_991204)
[ Status Polling Endpoint (Rate-Limited, Read-Optimized) ]
  ├── If State == "PROCESSING": Returns HTTP 200 + Retry-After: 60
  └── If State == "SUCCEEDED": Returns HTTP 200 + `result_url` (Signed S3 URL)
                       │
                       ▼ (Optional Asynchronous Push Callback)
[ Webhook Notification Dispatcher ]
  └── Emits HMAC-Signed `tax.statement.completed` to Registered Partner Webhook

Criteria and weights

Comparison

Result

Option C is selected. HTTP 202 Accepted immediately releases connection threads; Retry-After prevents polling storms; dual polling and webhook delivery support both mobile and automated partner consumers.

Required Mechanisms

1. Job Initiation Contract (HTTP 202 Accepted) [MC-JI-01]

• Client Request:

  POST /v1/tax-statements/jobs HTTP/1.1
  Host: api.bank.internal
  Authorization: Bearer <token>
  Content-Type: application/json
  Idempotency-Key: idemp_tax_2026_8812
  
  {
    "tax_year": 2025,
    "account_id": "acc_wealth_881204",
    "delivery_channel": "PORTAL_DOWNLOAD"
  }
  

• Server Response:

  HTTP/1.1 202 Accepted
  Location: https://api.bank.internal/v1/tax-statements/jobs/job_tax_991204
  Retry-After: 30
  Content-Type: application/json
  
  {
    "job_id": "job_tax_991204",
    "status": "PENDING",
    "created_at": "2026-09-15T14:22:00Z",
    "estimated_duration_seconds": 180
  }
  

2. Polling Status Protocol & Pacing [MC-PS-01]

• Status Endpoint: GET /v1/tax-statements/jobs/{job_id}.
• Pacing Rules:
  • Response includes mandatory Retry-After header specifying seconds until next poll.
  • Recommended backoff schedule: Initial poll: 30s -> Second poll: 60s -> Subsequent polls: 120s.
  • If client ignores Retry-After and polls within < 5 seconds, gateway returns HTTP 429 Too Many Requests.

3. Job Lifecycle State Machine [MC-SM-01]

• States: PENDING -> PROCESSING -> SUCCEEDED / FAILED / CANCELLED.
• Invariants:
  • Terminal states (SUCCEEDED, FAILED, CANCELLED) are immutable. Once terminal, state never changes.
  • Processing timeout ceiling: If a job remains in PROCESSING for > 15 minutes, an automated reaper marks it FAILED with error ERR_JOB_EXECUTION_TIMED_OUT.

4. Terminal Result Delivery & Retention [MC-RD-01]

• Successful Job Response:

  HTTP/1.1 200 OK
  Content-Type: application/json
  
  {
    "job_id": "job_tax_991204",
    "status": "SUCCEEDED",
    "completed_at": "2026-09-15T14:24:12Z",
    "result": {
      "download_url": "https://vault.bank.internal/exports/tax-2025-acc8812.pdf?token=...",
      "checksum_sha256": "7f8e3a2d4c1b9a0e6f5d8c3b2a1e0f9d8c7b6a5e4d3c2b1a0f9e8d7c6b5a4e3d",
      "expires_at": "2026-09-16T14:24:12Z"
    }
  }
  

Artifact Retention: Generated statements remain downloadable for

24 hours, after which the signed download URL expires.

Invariants and Contracts

Mandatory Immediate Socket Termination [INV-ASYNC-01]
  Job initiation requests must return HTTP 202 within 100 milliseconds.
  Holding client connection sockets open while waiting for background processing to complete is prohibited.

Enforced Retry-After Header Pacing [INV-ASYNC-02]
  All non-terminal status polling responses must include an authoritative `Retry-After` header.
  Clients violating polling intervals are throttled with HTTP 429.

Terminal State Immutability [INV-ASYNC-03]
  Once a job transitions to SUCCEEDED, FAILED, or CANCELLED, its state and metadata are frozen.
  Re-running or mutating terminal job records is strictly forbidden.

Explicit Unknowns

• S3 presigned URL generation latency when 45,000 jobs complete simultaneously during tax season peak (G-1).
• Webhook partner endpoint receiver downtime when delivering completion callbacks to external accounting software (G-2).

Traceability

Verification

No validator was supplied, so no command was run.

Reviewer self-check against async operation design standards:

• Thread Safety: PASS. Immediate HTTP 202 response terminates connection in < 15 ms.
• Pacing Rigor: PASS. Enforces Retry-After header and rate-limits aggressive polling.
• State Machine: PASS. Irreversible transitions with 15-minute execution reaper timeout.
• Markdown Hygiene: PASS. Native Markdown syntax strictly adheres to rule_markdown.md.

Open Decisions

• DEC-ASYNC-01: Elena Rostova to determine whether Server-Sent Events (SSE) should be offered as an alternate completion channel for web browser users (Owner: Elena Rostova).

Next steps

1. Marcus Vance provisions Redis cluster for job status metadata and token-bucket polling rate limiters.
2. Platform Engineering implements the HTTP 202 polling filter and Kafka job queuing workers.
3. Conduct staging resilience drill submitting 5,000 concurrent jobs to verify zero API gateway connection pool exhaustion.