The context wall in legal AI

Context engineering has become one of the most talked-about concepts in legal technology.

Within that broader discussion sits a narrower, more technical idea: the context window, essentially the short-term memory of a large language model (LLM).

The idea of a context window seems straightforward. As LLMs are built with larger context windows, they’ll be able to take in more information at once – and more information, in theory, should produce better outcomes.

Unfortunately, things are not so straightforward. It is increasingly clear that a bigger context window does not automatically mean better reasoning or better results. That’s because of what might be called “the context wall”: the point past which adding more content stops helping and can start hurting – either because there is simply too much material for even a very large window to hold, or because not everything inside the window is weighted evenly by the model.

Because that wall exists, something always has to decide what actually reaches the model – which documents, which excerpts, in what order. Left ungoverned, that decision defaults to whatever retrieval logic a vendor happens to have built in, rather than to what the matter actually requires, and firms end up with answers generated by reasoning over a fraction of the available knowledge base. To successfully surmount this context wall, firms must take an approach based on governed content delivery if they want optimal results from their AI.

When a window becomes a wall

The context window comes into play for a variety of tasks – for instance, when a fee earner asks an AI assistant to review a contract, summarise a matter, or identify relevant precedent. The model must then hold all of the relevant information in working memory at the same time.

That working memory is measured in tokens. Current frontier models advertise context windows ranging from 128,000 to one million tokens, numbers large enough to have persuaded many organisations that context constraints are effectively solved.

They are not. A standard active matter at a mid-sized firm, including correspondence, drafts, filed documents, research notes, and related emails, can easily represent several hundred thousand tokens of content. A single large transaction can easily run into the millions. No context window can hold that many tokens of information – at a certain point, it hits the wall.

Glossing over the middle bits

The specific way that information is contained and processed within the context window adds an additional challenge that needs to be reckoned with. Research into how LLMs actually use long contexts has found that not all information within a context window is treated equally. Much like the human mind, models tend to give more weight to information presented near the beginning and the end of what they are shown, while content placed in the middle is more likely to be overlooked or underweighted.

This matters enormously for document-heavy legal workflows. If a matter's documents are loaded in no particular order, the specific clause, admission, or piece of evidence a lawyer actually needs is just as likely to land in that attentional “dead zone” in the middle as at either end of the window.