ADR Writer: Architecture Decision Records

Wrote docs/adr/0042-direct-to-s3-uploads.md in MADR format. The decision, status, date and options come from your request. Four items you did not supply are recorded as open follow-ups, not filled in. The file-name slug is assumed from the title; rename it if your repository uses another pattern.

---
status: accepted
date: 2026-09-10
deciders: Architecture guild (minutes ARCH-GUILD-2026-37)
---

# 0042. Direct-to-S3 uploads with presigned URLs for tenant files

## Context and Problem Statement

Tenant file uploads of up to 2 GB are streamed through the API servers, so each upload holds an
API worker for the full transfer. In incident INC-2291 (2026-08-21), 40 concurrent uploads
exhausted the 64-worker pool and p99 API latency rose from 180 ms to 9.4 s. Upload traffic can
therefore take down unrelated API traffic.

How should tenant files reach storage without consuming API worker capacity?

## Decision Drivers

* API latency must not depend on the number or size of concurrent uploads (INC-2291).
* Files must pass malware scanning before any tenant can download them.
* Presigned URLs expire after 15 minutes.

## Considered Options

1. Direct-to-S3 presigned PUT and multipart upload
2. Dedicated upload service behind the gateway
3. Raise the API worker pool from 64 to 256

## Decision Outcome

Chosen option: **1. Direct-to-S3 presigned PUT and multipart upload**, because it is the only
option that removes upload bytes from the API path. The API issues a short-lived presigned URL;
the client sends the file straight to S3.

### Consequences

* Good: upload volume no longer consumes API workers, removing the failure mode of INC-2291.
* Good: no new long-running service to operate.
* Bad: the API no longer sees file bytes, so malware scanning must run after the object lands
  in S3 and before it becomes downloadable.
* Bad: clients must handle URL expiry after 15 minutes and resume or restart large multipart
  uploads.
* Neutral: upload authorization moves from request handling to URL issuance, which becomes the
  control point for who may upload what.

### Confirmation

This decision is implemented when an upload in progress does not occupy an API worker and a
newly uploaded object cannot be downloaded before its malware scan passes. The test that proves
both is a follow-up (F-1, F-2).

## Pros and Cons of the Options

### 1. Direct-to-S3 presigned PUT and multipart upload

* Good: bytes bypass the API servers entirely.
* Good: uses a managed storage service; no extra runtime.
* Bad: scanning and download gating must be built around S3 events.

### 2. Dedicated upload service behind the gateway

* Good: isolates uploads from the main API workers.
* Bad: one more service to build and run.
* Bad: still proxies every byte, so the capacity problem moves instead of disappearing.

### 3. Raise the API worker pool to 256

* Good: smallest code change.
* Bad: higher memory cost.
* Bad: the coupling between uploads and API latency remains; a larger burst repeats INC-2291.

## More Information

* Incident: INC-2291 (2026-08-21).
* Decision minutes: ARCH-GUILD-2026-37 (2026-09-10).
* Follow-ups, owners not yet assigned:
  * F-1 Download gating: objects stay non-downloadable until the malware scan passes.
  * F-2 Load test proving uploads do not occupy API workers.
  * F-3 Multipart threshold and client resume behaviour after URL expiry.
  * F-4 Cost impact of S3 requests and storage versus the current path.

Record check

No validator was supplied, so no command was run.

Not supplied, so not decided

• Who owns the malware scanning pipeline (F-1).
• The multipart size threshold (F-3).
• Whether existing uploads in flight need a migration path.
• Cost figures (F-4).

Next steps

1. Assign owners for F-1 to F-4 in your tracker and link them from the ADR.
2. Rename the file if your slug convention differs from 0042-direct-to-s3-uploads.md.
3. Commit the ADR before the code review, as planned.