LLM context compression

LLM context compression that keeps the original source accountable.

LLM context compression is the decision about what an agent sends into its next model call. Tokz selects relevant source material from tool output, retrieved documents, and transcripts, then makes the retained material checkable against the input.

import OpenAI from "openai";
import { withTokz } from "@tokz/openai";

const openai = withTokz(new OpenAI(), {
  apiKey: process.env.TOKZ_API_KEY!,
});

await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [
    { role: "user", content: "Which deployment failed?" },
    { role: "tool", tool_call_id: "call_1", content: deploymentJson },
  ],
});

Context is not one undifferentiated prompt

An agent request can contain instructions, conversation history, retrieved knowledge, and tool data. Compressing them with one blind ratio creates avoidable risk. Tokz supports protected and control context alongside compressible data so applications can choose what must remain intact.

Protected context

Keep rules, secrets, and application-owned requirements outside the compression decision.

Data context

Select from verbose tool output and retrieved material where reduction is measurable and reviewable.

Choose selection over truncation

Truncation is predictable but position-biased: a relevant error at the end of a log can disappear simply because it is late. Context compression can select non-contiguous spans that relate to the request, then preserve the original bytes on the structural path.

Keep prompt-cache economics in the decision

A smaller prompt is not automatically a cheaper prompt. Provider caches reward stable prefixes, so a policy that changes an already-cached prefix can cost more than the input tokens it removes. Measure cache reads and writes before changing a long-lived context layout.

Limitations

Selection cannot recover a fact absent from the source or prove that a downstream model will reason correctly. For free prose, Tokz returns extractive chunks with provenance, not an offsets-only response.

Continue with the implementation details