Backend Architect Pro

Backend design review — order processing (fictional)

Decision summary

Do not promise exactly-once end-to-end processing. Design for at-least-once delivery with idempotent state transitions, a transactional outbox, and reconciliation for uncertain payment outcomes. Treat 99.99% as an unvalidated target until service boundaries, dependencies, and failure budgets are defined.

Clarify first

• What is the authoritative order state and what transitions are legal?
• Which payment operations are idempotent, and what reference/key does the provider support? What is its timeout and reconciliation interface?
• What latency and throughput distribution, burst, tenant count, and regional requirements apply?
• What are RPO/RTO, data retention, privacy, audit, and failure-response obligations?
• Can fulfillment tolerate delay, duplicate messages, or reordering? Which downstream dependencies have quotas?

Core invariants

1. A client retry with the same idempotency key cannot create a second order.
2. Order and outbox event are committed atomically in one database transaction.
3. Payment authorization/capture status is not inferred from a client timeout.
4. Fulfillment begins only after the required authoritative payment state and policy checks.
5. Every state transition is auditable, bounded, and safe to retry.

API contract

Require an idempotency key for order creation; bind it to authenticated tenant/customer and request hash. Persist the key, result, and expiration policy. Return the original response for a matching retry; reject key reuse with a different payload. Include a stable order ID and explicit pending, confirmed, or needs-reconciliation status. Define validation errors, rate limits, request size, versioning, and correlation IDs. Do not claim a latency SLA before representative load testing.

Transaction and event options

In one database transaction, validate inventory policy, write the order state, and append an outbox record. A relay publishes outbox records and marks delivery progress; it may publish duplicates, so consumers deduplicate by event ID and enforce idempotent transitions. If event ordering matters, partition by order ID and test consumer recovery. Monitor oldest outbox age and backlog; establish replay and dead-letter procedures.

Payment uncertainty

Use bounded connect/read timeouts and provider-supported idempotency keys. If the request times out after submission, mark the payment outcome unknown and query/reconcile by provider reference before retrying. Never blindly submit another charge. Use a state machine such as created → payment_pending → paid | declined | reconciliation_required → fulfillment_pending → fulfilled. Send manual review tasks for conflicting or prolonged states.

Security and resilience

Authenticate callers, authorize tenant and order ownership on every operation, validate payloads, use scoped service identities, store secrets in an approved secret manager, and redact payment and personal data from logs. Apply quotas, circuit breakers, concurrency limits, and backpressure. Keep the API available for safe status reads during payment-provider degradation while refusing unsafe duplicate actions.

Validation and operations

Test concurrent duplicate requests, key/payload mismatch, database commit failure, event duplicate/reorder, relay restart, provider timeout before and after side effect, reconciliation, DLQ replay, tenant isolation, and migration rollback. Measure request latency/errors, database saturation, outbox age, consumer lag, payment unknown-state duration, reconciliation backlog, and order completion.

Next steps

Agree on RPO/RTO and service-level objectives, confirm provider semantics from its current documentation, prototype the idempotency/outbox path, and run failure-injection and load tests. Exactly-once business outcomes are an invariant to approximate with state and reconciliation, not a delivery guarantee. No system was inspected or deployed in this example.