Context window

The context window is how much text a language model can process at once in a single request. Everything goes in it: the instruction, any documents you supply, the conversation so far, and the answer itself. When the window is full, the oldest part falls out.

How is the context window measured?

In tokens, not words. A token is a fragment of text, often a syllable or a short word. A rough rule for English is one token per four characters. German compound words split into several tokens, so the same content needs more window in German than in English.

Large windows are now the norm. That moves the limit rather than removing it, and it moves it at a price: every token in the window is paid for and processed on every request.

Why does the context window matter?

It decides how large a task may be. A forty-page contract fits in a large window; two hundred contracts do not. That is exactly where RAG becomes necessary: instead of supplying everything, you search and insert only what fits the question.

It also drives cost. A system that sends a long instruction plus the whole conversation on every request pays for that text a thousand times over. Whatever stays constant belongs in fine-tuning rather than in every individual call.

And it affects quality. Models do not use a window evenly: what sits at the beginning and end carries more weight than what sits in the middle. A window filled to the brim is therefore not an advantage, and is often the reason an instruction gets ignored.

What follows in practice

Three rules hold up well.

Put in less, but the right things. Good retrieval beats a large window almost every time, because it improves the ratio of signal to noise instead of just raising the volume.

Put what matters at the edges. Instructions and the actual question belong at the front or the back, not sandwiched between twenty pages of appendix.

Trim conversation history deliberately. In an agentic workflow with many steps the history otherwise grows unnoticed until costs climb and the original instruction drops out of the window.

A worked example

Say an assistant answers service requests. The instruction is thorough and runs to roughly 1,500 tokens. Retrieval supplies five passages of about 400 tokens each, so 2,000. The request itself is short, say 100 tokens. The answer runs to 300.

That puts a single call at around 3,900 tokens, of which 3,500 is input incurred again on every call. The actual question accounts for less than three percent.

Two things become visible. First, the lever is the instruction and the supplied passages, not the answer length. Cut from five passages to three while holding retrieval quality and you save a fifth of the cost with no loss in output.

Second, a conversation with ten follow-ups does not cost ten times as much but more, because every prior exchange gets resent. This is exactly where summarising the history beats appending it in full.