Trimwise Compress

Decision

Trimwise is useful here because the current code selects by position, even when relevant sections are later in the documents. Local Python keeps the supplied text in your process. This change uses query-aware word matching; it does not need an embedding model.

Project change

Install the tested library version

python -m pip install "trimwise==0.7.0"

Replace context assembly with this function and pass the question to it. Create the Trimmer once and reuse it:

from trimwise import ContextSource, TrimConfig, Trimmer

TRIMMER = Trimmer(TrimConfig(token_encoding="o200k_base"))

def build_context(documents, question, budget):
    sources = [ContextSource(text=text, prefix=f"[{source_id}]\n") for source_id, text in documents]
    return TRIMMER.trim_context(sources, limit=budget, query=question, separator="\n\n").text

context = build_context(documents, question, 180)

The limit includes source labels and separators. Keep the existing retrieval and model call. Count the complete assembled prompt with the target model's tokenizer before sending it.

Measured result on this fixture

The three synthetic documents contain 6,114 tokens of source text. Both methods were run with the same 180-token evidence budget and o200k_base tokenizer.

The details checked are the release change, stale key, reload step, conditional rollback, and requirement to keep signature validation enabled. This checks this fixture; it is not a benchmark or a guarantee about answer quality.

Actual returned context

[release-notes]

# Release notes

Routine checks covered dashboards, deployment schedules, and team handovers. The team recorded status updates and reviewed the next maintenance window.

[…omitted…]

Routine checks covered dashboards, deployment schedules, and team handovers. The team recorded status updates and reviewed the next maintenance window.

## Change linked to the failures

Release 2.4 changed how webhook signing keys are rotated. Webhook failures began after that release; retries alone did not fix them.

[incident-review]
[…omitted…]

## Root cause

The webhook worker kept an old signing key in memory. New requests were checked against the stale key, so signature validation failed.

[recovery-plan]

## Approved fix and rollback

Reload signing keys before enabling the webhook worker. Roll back to release 2.3 if signature validation failures continue after the reload. Do not disable signature validation.

Reproduce the check

After installing Trimwise, save this as demo.py and run python demo.py. It includes the original method, replacement, synthetic documents, and checks. Initial installation and tokenizer assets may need a download; the document processing is local.

"""Reproduce a context-assembly change with synthetic incident documents."""

import tiktoken
from trimwise import ContextSource, TrimConfig, Trimmer

EVIDENCE_BUDGET = 180
QUESTION = (
    "Which release caused the webhook failures, what failed, "
    "and what is the fix and rollback requirement?"
)
TOKENIZER = tiktoken.get_encoding("o200k_base")
TRIMMER = Trimmer(TrimConfig(token_encoding="o200k_base"))

def original_context(documents, budget):
    separator = "\n\n"
    separator_tokens = len(TOKENIZER.encode(separator)) * (len(documents) - 1)
    per_source = (budget - separator_tokens) // len(documents)
    excerpts = []
    for source_id, text in documents:
        labeled_text = f"[{source_id}]\n{text}"
        excerpts.append(TOKENIZER.decode(TOKENIZER.encode(labeled_text)[:per_source]))
    return separator.join(excerpts)

def build_context(documents, question, budget):
    sources = [ContextSource(text=text, prefix=f"[{source_id}]\n") for source_id, text in documents]
    return TRIMMER.trim_context(sources, limit=budget, query=question, separator="\n\n").text

BACKGROUND = (
    "Routine checks covered dashboards, deployment schedules, and team handovers. "
    "The team recorded status updates and reviewed the next maintenance window.\n\n"
) * 80
DOCUMENTS = [
    (
        "release-notes",
        "# Release notes\n\n" + BACKGROUND + "## Change linked to the failures\n\n"
        "Release 2.4 changed how webhook signing keys are rotated. "
        "Webhook failures began after that release; retries alone did not fix them.",
    ),
    (
        "incident-review",
        "# Incident review\n\n" + BACKGROUND + "## Root cause\n\n"
        "The webhook worker kept an old signing key in memory. "
        "New requests were checked against the stale key, so signature validation failed.",
    ),
    (
        "recovery-plan",
        "# Recovery plan\n\n" + BACKGROUND + "## Approved fix and rollback\n\n"
        "Reload signing keys before enabling the webhook worker. "
        "Roll back to release 2.3 if signature validation failures continue after the reload. "
        "Do not disable signature validation.",
    ),
]
EXPECTED_DETAILS = [
    "Release 2.4 changed how webhook signing keys are rotated.",
    "The webhook worker kept an old signing key in memory.",
    "Reload signing keys before enabling the webhook worker.",
    "Roll back to release 2.3 if signature validation failures continue after the reload.",
    "Do not disable signature validation.",
]

before = original_context(DOCUMENTS, EVIDENCE_BUDGET)
after = build_context(DOCUMENTS, QUESTION, EVIDENCE_BUDGET)
before_tokens = len(TOKENIZER.encode(before))
after_tokens = len(TOKENIZER.encode(after))
before_details = sum(detail in before for detail in EXPECTED_DETAILS)
after_details = sum(detail in after for detail in EXPECTED_DETAILS)
assert before_tokens <= EVIDENCE_BUDGET
assert after_tokens <= EVIDENCE_BUDGET
assert before_details == 0
assert after_details == len(EXPECTED_DETAILS)
assert all(f"[{source_id}]" in after for source_id, _ in DOCUMENTS)

print(f"Before: {before_tokens} tokens; {before_details}/5 fixture details")
print(f"After: {after_tokens} tokens; {after_details}/5 fixture details")
print("\nTrimwise context:\n" + after)

You can also use the free public API

If you’d rather not add Trimwise to your application, we can use its free public API instead. There’s no account or API key to set up, and your app can use its existing HTTP client. This option sends the supplied text to the service; local Python is available when the text needs to stay in your process.

Here’s the same context builder using HTTPx, which this example project already uses. You do not need the Trimwise library for this path. Use the same documents and question from the example above:

import httpx

API_BASE = "https://trimwise.aatbit.com"
EVIDENCE_BUDGET = 180

def build_context_via_api(client, documents, question):
    response = client.post(
        f"{API_BASE}/api/v1/context",
        json={
            "sources": [
                {"text": text, "prefix": f"[{source_id}]\n"} for source_id, text in documents
            ],
            "limit": EVIDENCE_BUDGET,
            "unit": "tokens",
            "strategy": "auto",
            "query": question,
            "separator": "\n\n",
        },
    )
    response.raise_for_status()
    return response.json()

# DOCUMENTS and QUESTION are the same fixtures used in the Python example.
with httpx.Client(timeout=httpx.Timeout(15.0, connect=5.0)) as client:
    result = build_context_via_api(client, DOCUMENTS, QUESTION)
context = result["text"]

The deployed API v0.2.0 returned the same excerpt shown above. Its response metadata was:

{
  "input_count": 6114,
  "output_count": 178,
  "limit": 180,
  "unit": "tokens",
  "strategy": "lexical"
}

I also checked the returned text with o200k_base: it was 178 tokens, kept all three source labels, and retained the same five incident details. The excerpt and the final assembled prompt still need separate budget checks.

Check the service’s current request limits once at GET /api/v1/limits when setting up the integration. This example uses a 15-second HTTP timeout and lets the application handle HTTP errors. If you add retries for busy or rate-limited responses, respect Retry-After and cap both attempts and wait time.

Next steps

Check representative documents from your project, including exceptions and conflicting sources. Choose an explicit fallback when useful evidence does not fit. Keep instructions and answer space separate from the evidence limit; excerpts can still leave out information needed for an answer.