API Rate Limit Architecture Planner

Rate-Limiting, Penalty Box & Backoff Architecture

Multi-tenant B2B FinTech Payment Ledger & Reconciliation REST API

1. Executive Architecture Summary

• Primary Algorithm Selected: Cost-weighted Token Bucket (GCRA-equivalent) for all tenant and key quotas.
  • Growth tenants run end-of-day batch spikes, so the bucket must allow short bursts while holding a strict long-run average.
  • Cost weighting lets one bucket protect cheap and expensive endpoints alike.

  • Supplemented by:

Sliding Window Counter for POST /v1/auth/token and failed-login tracking, where precision matters more than burst tolerance.
-

Concurrency semaphores (Redis ZSET leases) plus

PostgreSQL advisory locks for reconcile and audit-export. A rate limit alone can't protect against long-running lock-holding requests.
- Fixed Window at AWS WAF only, as a cheap, coarse per-IP backstop.

• Enforcement Layer: Defense in depth across three layers.
  1.

AWS WAF + API Gateway: coarse per-IP rate-based rules and stage-level throttles. This is a flood backstop, not business logic.
2.

Kong Enterprise (primary enforcement): a custom rate-limit plugin that runs after key authentication and calls the Redis Lua script below. The built-in rate-limiting-advanced plugin handles the IP and auth-endpoint sliding windows.
3.

Fastify middleware: concurrency leases, the per-ledger advisory lock, and idempotency checks, because only the service knows which ledger a request touches.

• State Store & Synchronization: Redis Enterprise Cluster with atomic Lua scripts via EVALSHA.
  • All keys for one tenant share a hash tag ({t:<tenant_id>}) so multi-key scripts stay single-slot.
  • Time comes from redis.call('TIME'), never from the gateway clock, so there is no skew across Kong nodes.
• Failure Mode Strategy: Deliberately split by endpoint risk.

flowchart TD
    A[Client request] --> B[AWS WAF: per-IP rate rule + blocklist]
    B -->|blocked| X1[403 / 429 at edge]
    B --> C[API Gateway stage throttle]
    C --> D[Kong: authenticate API key -> tenant_id, tier]
    D --> E{Penalty box active?}
    E -->|yes| X2[429 + Retry-After + X-Penalty-Box-Status: Active-Lockout]
    E -->|no| F[Redis Lua: token bucket, cost = endpoint weight]
    F -->|insufficient tokens| G[Record strike -> maybe trip box] --> X3[429 + Retry-After]
    F -->|ok| H{Heavy endpoint?}
    H -->|yes| I[Concurrency lease + PG advisory lock]
    I -->|denied| X4[429/409 + Retry-After]
    I -->|granted| J[Fastify handler]
    H -->|no| J
    J --> K[Response + RateLimit headers]

---

2. Multi-Tier Rate Limit & Quota Matrix

Limits are in

points. A standard request costs 1 point, so the "req/min" figures equal points/min for standard traffic. All values are starting defaults to tune from production telemetry.

Tenant-level and key-level buckets are both checked, and the stricter one wins.

---

3. Endpoint Cost & Resource Protection Rules

• Standard CRUD Operations (1 Point): GET on ledgers/entries/accounts and ordinary POST entry creation.

  • Mutating POSTs require an Idempotency-Key, so client retries after a 429 or timeout can never double-post.
  • No concurrency lock is needed beyond the normal DB pool.

Compute-Intensive / Mutation Operations (5–10 Points): POST /v1/ledgers/reconcile is

10 points. It triggers heavy PostgreSQL locking, so token cost alone is insufficient. Add all of the following:

Per-ledger mutual exclusion: pg_try_advisory_xact_lock(hashtext(tenant_id || ledger_id)). A second reconcile on the same ledger gets 409 Conflict with Retry-After (never queue it behind a lock).

Per-tenant concurrency lease limit: Sandbox 1, Growth 2, Enterprise 8 (contractual). Leases live in a Redis ZSET with a TTL equal to the max job duration plus a safety margin, so a crashed worker can't leak a slot.

Server-side guards: SET LOCAL lock_timeout = '2s' and statement_timeout. Abort rather than stack up lock waits.

Preferred shape: accept the request, return 202 Accepted with a job URL, and run reconciliation on a worker pool. Rate limits then control submission, and the pool controls execution.

Bulk / Async Export & AI Generation (25+ Points): GET /v1/reports/audit-export is

25 points (cost scales to 50 for date ranges over 90 days).

• Concurrency lock: Sandbox 1 / Growth 1 / Enterprise 3 concurrent exports per tenant.

Async pattern: return 202 and a job ID, generate the CSV on a worker, and write it to object storage. Serve a short-lived signed URL, so large CSVs never stream through the API tier.

Safety rails: read-replica only, row and size caps per export, and no more than N exports per tenant per hour (Sandbox 2 / Growth 10 / Enterprise 60).

• Not for GET /audit-export retries: clients poll the job endpoint (cost 1) instead of re-requesting.

• Credential Endpoint (POST /v1/auth/token): has its own limiter, independent of tier.

  • Counted: only failed attempts count toward lockout, so a legitimate client refreshing tokens isn't punished.

Response: a uniform response and latency for "unknown client" versus "bad secret", so responses can't be used to enumerate accounts.

Credential stuffing detection: many distinct client_ids or subjects from one IP, or one subject from many IPs. Either pattern feeds the Level 3 criteria below.

---

4. Penalty Box & Progressive Abuse Mitigation Architecture

Trigger Conditions: A

strike is recorded each time a request is rejected for lack of tokens. This means the client sent a request before its Retry-After elapsed.

• Sandbox/Growth: 20 strikes within 60 s trips the box.

• Enterprise: 100 strikes within 60 s.

• Auth endpoint: 5 failed attempts per subject in 15 min trips the box immediately for that subject, and 20 failures per IP in 15 min trips it for that IP.

• A client that honors Retry-After never accumulates strikes.

• Escalation Tiers: Offense count is tracked per scope over 24 h. Each box trip increments it.

  • Level 1 (Soft Cool-Off): first trip.
    • Duration: 5 min.
    • Scope: the offending api_key_id only. Other keys on the tenant are unaffected.

Behavior: requests are rejected at Kong with 429 and no token evaluation, so no Redis bucket work and no further strikes.
- Recovery: automatic when the TTL expires. The offense counter stays for 24 h.

• Level 2 (Hard Quarantine): second trip within 24 h.
  • Duration: 60 min.

Edge block: key added to a Kong deny cache. For unauthenticated or auth abuse, the source IP also goes into an AWS WAF IP set (TTL-managed).
-

Alert: webhook and email to the tenant's technical contacts, plus an internal notification (Slack/PagerDuty to security and support). The message includes the key, the offending endpoints, and a pointer to the backoff guidelines.
- Recovery: automatic on expiry, or early release by support.

• Level 3 (Permanent Revocation / WAF Ban): third trip within 24 h,

or a credential-stuffing signature (for example 500 or more failed auths across 50 or more distinct subjects from one IP or ASN in 10 min).
- Unauthenticated or auth abuse: WAF IP ban (24 h, renewable by the security team) and security ticket.
-

Authenticated tenant keys: key

suspended (not deleted) pending human review. Do not auto-permanently revoke

Enterprise keys. Their SLA and payment flows make this a mandatory human-in-the-loop decision, so Level 3 for Enterprise pages the on-call and the account owner.
- Recovery: manual only, with an audit-log entry.

State Key Schema: {...} is the Redis Cluster hash tag, which keeps all keys for one scope in one slot. Scope identifiers are hashed or opaque IDs, never raw secrets.

---

5. HTTP Response Headers & Retry-After Specification

These use the three-header form requested. The IETF draft (draft-ietf-httpapi-ratelimit-headers) is still a draft, and recent revisions consolidate it into structured RateLimit / RateLimit-Policy fields. Pin the draft version you implement, and consider emitting the newer form alongside it once your SDKs parse it.

Standard Successful Response (200 OK)

HTTP/1.1 200 OK
RateLimit-Policy: 3000;w=60
RateLimit-Limit: 3000
RateLimit-Remaining: 2412
RateLimit-Reset: 18

RateLimit-Remaining is in

points. RateLimit-Reset is delta-seconds until the bucket would be full, never an absolute timestamp.

Throttled Response (429 Too Many Requests)

HTTP/1.1 429 Too Many Requests
Content-Type: application/problem+json
Retry-After: 12
RateLimit-Policy: 3000;w=60
RateLimit-Limit: 3000
RateLimit-Remaining: 0
RateLimit-Reset: 12
X-Penalty-Box-Status: Warning

• X-Penalty-Box-Status is Warning while the client is accumulating strikes, and Active-Lockout once boxed.
• For Active-Lockout, Retry-After is the remaining box time. For boxes of an hour or more, it may be sent as an HTTP-date instead (Retry-After: Wed, 14 Oct 2026 15:00:00 GMT).
• Retry-After is the

server's minimum. It should include a small server-side random component (0-10%) so many clients throttled in the same second don't return in the same second.

Standardized Error Payload (application/problem+json)

{
  "type": "https://api.yourdomain.com/errors/rate-limit-exceeded",
  "title": "Too Many Requests",
  "status": 429,
  "detail": "Quota for tier 'growth' exhausted (3000 points/min). POST /v1/ledgers/reconcile costs 10 points. Capacity refills at 50 points/second; retry in 12 seconds.",
  "instance": "/v1/ledgers/reconcile",
  "retry_after_seconds": 12,
  "recommended_backoff": "exponential_with_full_jitter",
  "quota_tier": "growth",
  "penalty_strikes_remaining": 14,
  "request_id": "req_01HV..."
}

• Do not include internal key names, Redis details, or other tenants' information in detail.
• For Active-Lockout, set type to .../errors/penalty-box-active and add penalty_level and lockout_ends_at.
• Concurrency denials use .../errors/concurrency-limit-exceeded.

---

6. Reference Implementation Logic (Atomic State Check)

Called from Kong for each authenticated request, with KEYS all sharing one hash tag. It returns {status, remaining_points, retry_after_seconds, box_level, strikes_remaining}, where status is 1 (allowed), 0 (throttled), or -1 (penalty box).

-- KEYS[1] = rl:{t:T}:tb:<key_id>        token bucket (hash)
-- KEYS[2] = pb:{t:T}:box:<key_id>       active penalty box (value = level)
-- KEYS[3] = pb:{t:T}:strikes:<key_id>   strike counter
-- KEYS[4] = pb:{t:T}:off:<key_id>       offense counter (24h)
-- ARGV[1] = refill rate (points/sec)     ARGV[2] = bucket capacity (burst)
-- ARGV[3] = request cost (points)        ARGV[4] = strike window (s)
-- ARGV[5] = strike threshold             ARGV[6] = L1 box (s)
-- ARGV[7] = L2 box (s)                   ARGV[8] = L3 box (s)
-- ARGV[9] = offense TTL (s)

local rate      = tonumber(ARGV[1])
local capacity  = tonumber(ARGV[2])
local cost      = tonumber(ARGV[3])
local win       = tonumber(ARGV[4])
local threshold = tonumber(ARGV[5])

local t = redis.call('TIME')
local now_ms = tonumber(t[1]) * 1000 + math.floor(tonumber(t[2]) / 1000)

-- 1. Penalty box check: reject without touching the bucket or adding strikes
local box_ttl = redis.call('PTTL', KEYS[2])
if box_ttl > 0 then
  local lvl = tonumber(redis.call('GET', KEYS[2]))
  return {-1, 0, math.ceil(box_ttl / 1000), lvl, 0}
end

-- 2. Refill the bucket
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(b[1])
local ts     = tonumber(b[2])
if tokens == nil then
  tokens = capacity
  ts = now_ms
end
tokens = math.min(capacity, tokens + (math.max(0, now_ms - ts) / 1000) * rate)
local bucket_ttl = math.ceil(capacity / rate) * 2

-- 3. Allow
if tokens >= cost then
  tokens = tokens - cost
  redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now_ms)
  redis.call('EXPIRE', KEYS[1], bucket_ttl)
  return {1, math.floor(tokens), 0, 0, threshold}
end

-- 4. Throttle: persist the refill, compute Retry-After, record a strike
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now_ms)
redis.call('EXPIRE', KEYS[1], bucket_ttl)
local retry_after = math.ceil((cost - tokens) / rate)

local strikes = redis.call('INCR', KEYS[3])
if strikes == 1 then redis.call('EXPIRE', KEYS[3], win) end

if strikes >= threshold then
  local offenses = redis.call('INCR', KEYS[4])
  redis.call('EXPIRE', KEYS[4], tonumber(ARGV[9]))
  local level = math.min(offenses, 3)
  local dur = tonumber(ARGV[5 + level])        -- ARGV[6], [7], [8]
  redis.call('SET', KEYS[2], level, 'EX', dur)
  redis.call('DEL', KEYS[3])
  -- Caller (Kong plugin) inspects level: 2 -> alert + edge deny, 3 -> security escalation
  return {-1, 0, dur, level, 0}
end

return {0, math.floor(tokens), retry_after, 0, threshold - strikes}

Concurrency lease (heavy endpoints), pseudo-code:

acquire(tenant, kind, max_concurrent, lease_ttl_s):
  EVAL atomically:
    ZREMRANGEBYSCORE cc:{t:T}:kind -inf now          -- drop expired leases
    if ZCARD cc:{t:T}:kind >= max_concurrent: return DENY (retry_after = earliest lease expiry - now)
    ZADD cc:{t:T}:kind (now + lease_ttl_s) lease_id
    return OK(lease_id)
release(lease_id): ZREM in a finally block; the TTL covers crashes

Dispatch and failure handling:

on Redis error/timeout (>20 ms):
  if endpoint in {reconcile, audit-export, auth/token}: return 503 + Retry-After: 15  (fail-closed)
  else: apply local degraded limiter (25% of tier limit per Kong node), tag metric rl_degraded=1

Load the script once with SCRIPT LOAD and call EVALSHA. Handle NOSCRIPT by reloading it.

---

7. Client SDK Backoff & Jitter Guidelines

• Exponential Backoff Formula (Full Jitter):

  sleep = random_uniform(0, min(cap, base × 2^attempt))
  base = 0.5 s, cap = 30 s, max_attempts = 5
  

  Full jitter spreads retries across the whole interval, which matters for Growth tenants whose end-of-day batches all start at the same time.

• Respecting Retry-After Precedence:

  1. If the response carries Retry-After, treat it as a

hard floor. Parse delta-seconds, or an HTTP-date relative to the response Date header, never the local clock.
2. Wait max(Retry-After, backoff_sleep), plus a small random addition (0 to 10% of the wait) to avoid herd returns.
3. Never retry sooner than Retry-After. Early retries are exactly what generate penalty-box strikes.
4. If X-Penalty-Box-Status: Active-Lockout,

stop all retries for that key until Retry-After elapses. Surface the error to the caller instead of looping.
5. For 429 on /auth/token, retry at most 2 times and never rotate through other credentials.
6. Retry only idempotent requests, or mutating requests that carry an Idempotency-Key. POST /v1/ledgers/reconcile must always send one.
7. On 409 from a reconcile or export concurrency conflict, poll the existing job. Don't re-submit.
8. Don't retry after max_attempts, and when the total wait would exceed the caller's deadline, fail fast with the last 429 and its request_id.
9. Expose RateLimit-Remaining to the application so batch jobs can pace themselves proactively, slowing before the bucket empties instead of reacting to 429s.

---

Notes for implementation

Kong token-bucket gap: the bundled Kong plugins don't do cost-weighted token buckets with a penalty box. This design assumes a small custom Kong plugin that calls the Lua script above, with rate-limiting-advanced handling the IP and auth windows.

Rollout: run in shadow mode for 1-2 weeks (log what would be throttled, enforce nothing), then enforce. Review the numbers against real Growth end-of-day batch peaks before enabling the penalty box.

Observability: alert on 429 rate per tier, penalty-box trips by level, rl_degraded fail-open events, and concurrency-lease denials.

If you'd like, I can turn this into a Docs document, or write a Kong custom plugin skeleton and a Fastify concurrency-lease middleware next.