Beyond prompts: mastering context engineering and avoiding “context rot” in LLMs

If you’re building with AI, you’ve probably heard the term “prompt engineering.” It’s the art of crafting the perfect query to get the desired output from a Large Language Model (LLM). But once you move from simple Q&A bots to complex, multi-step AI agents, that approach stops being enough. Context engineering is the skill that takes its place.

Context engineering is the art and science of filling the LLM’s “context window” (its short-term memory) with the right information at the right time. Think of it less like whispering a single magic spell and more like being a librarian who pulls the exact books a researcher needs for each step of their project.

But there is a catch: Context Rot. New research shows that as you feed more information into an LLM’s context window, its performance doesn’t just get slower; it gets worse. Context engineering is how you fight it off.

What is context engineering?

The term “context engineering” has gained traction because it more accurately describes the core skill needed for building sophisticated AI. As Shopify CEO Tobi Lutke and AI researcher Andrej Karpathy have pointed out, we’re moving beyond single, short prompts. We’re building systems that need to handle instructions, external knowledge, and feedback from tools over extended interactions.

Karpathy suggests thinking of an LLM as a new kind of operating system.

• The LLM is the CPU, the core processor.
• The context window is the RAM, the volatile working memory.

Just like a computer’s RAM, the context window is limited. If you try to stuff too much into it, or fill it with irrelevant junk, the “CPU” gets overwhelmed and performance degrades. Context engineering is the discipline of managing this RAM effectively.

To do this, developers rely on four strategies:

1. Writing context. This means saving information outside the context window to help an agent perform a task. It’s like taking notes. This can be a short-term “scratchpad” for the current session or long-term “memory” that persists across many sessions.
2. Selecting context. This involves pulling the right information into the context window at the right moment. This could mean retrieving relevant tools, pulling from the scratchpad or long-term memory, or fetching knowledge from a database.
3. Compressing context. To stay within token limits and keep things efficient, it’s often necessary to shrink the context. This can be done by summarizing conversations or trimming irrelevant details so that only the most critical tokens are used.
4. Isolating context. This strategy involves splitting context up. In a multi-agent system, for example, each agent can operate in parallel with its own isolated context window. This allows the overall system to process far more information than a single agent could.

The silent performance killer: context rot

So why is managing the context window so important? A recent Chroma research report, “Context Rot: How Increasing Input Tokens Impacts LLM Performance,” reveals a critical vulnerability in today’s models.

The researchers found that as the amount of input text (the context) increases, an LLM’s ability to perform tasks reliably degrades. This happens even when the total amount of information is well within the model’s advertised context window, like 1 million tokens.

Here is what the report found:

• Models struggle with long conversations. Even the most advanced models have trouble reasoning over long conversation histories. The more you add, the harder it is for the model to find the relevant information.
• Ambiguity makes it worse. When a query is slightly ambiguous, the challenge is compounded. Models get more easily confused and distracted by irrelevant information in a large context.
• Distractors are deadly. The presence of information that is topically related but not the correct answer (a “distractor”) harms performance, especially with long inputs.
• LLMs aren’t reliable computers. Unlike a traditional computer program that will execute the same way every time, an LLM’s performance is not consistent across different input lengths. A task that works perfectly with a 500-token input might fail completely with a 10,000-token input, even if the core information is the same.

This means you can’t just dump your entire chat history or a massive document into the prompt and expect the best results. Your “1 million token” context window doesn’t guarantee 1 million tokens of reliable reasoning.

How to build better with context engineering

To build reliable AI applications, you have to be deliberate about what you put in the context window. Here’s how the four pillars of context engineering directly address the problem of context rot:

• By Writing context to a scratchpad or memory, you avoid needlessly bloating the context window with information that isn’t immediately relevant.
• Through careful Selection (for instance, using Retrieval Augmented Generation or RAG), you pull in only the most relevant facts, tools, or memories needed for the current step and leave the noise out.
• Compression techniques like summarization can distill long histories into a more potent, token-efficient format.
• Isolation in multi-agent systems allows you to break down a large, complex problem into smaller sub-problems, each with its own focused and manageable context.

Frameworks like LangChain and its orchestration library LangGraph are designed to make these strategies easier to implement. They provide built-in tools for managing memory, state, and the flow of information between the different components of your agent, so you can engineer context with precision.

The takeaway

As developers, we need to shift our thinking. The future of AI isn’t just about bigger models and larger context windows; it’s about using them more intelligently. That is what context engineering is for. Master writing, selecting, compressing, and isolating context, and you can build applications that are more powerful and more reliable, with none of the performance pitfalls of context rot.