LLMs for agents

Structured I/O, context engineering, and not stuffing the window

The typist in a loop

Source

A reader’s guide to part of CS 329Z Lecture 2 (Diyi Yang, Michael Ryan and John Yang; Engineering AI Agents, Stanford, autumn 2026). The outline and the pointers come from that lecture; the wording on this page is ours. The course is also listed under Resources.

The last part of Lecture 2 is the bridge back to Lecture 1. You now have a next-token model, an architecture, a training recipe, and an inference engine. An agent is that model in a loop, talking to tools, files, and other components. Two practical problems show up immediately: the parts need a contract for how they talk, and the model’s context window is a scarce, slightly unreliable working memory.

Everyday example

Picture a busy office again. The clerk from Lecture 1 can write letters, but the office only works if the forms have a fixed layout (structured input and output), if you do not dump the entire filing cabinet on the desk (context rot), and if you add new papers to the end of the pile so nobody has to reshuffle what was already in hand (the KV cache). A short list of tools — a folder, a pen, a terminal — often beats a wall of rarely used gadgets.

Structured input and output

Zaharia and colleagues, in the BAIR note we already used on Compound AI systems, put it plainly: more and more results come from compound systems, not from a single model call. Those systems need a shared contract for how the parts communicate.

Figure 1: From a single model to a compound system that needs a contract between its parts — CS 329Z Lecture 2 (Diyi Yang, Michael Ryan, John Yang), slide p. 152. Original: Compound AI Systems (Zaharia et al., BAIR, 18 February 2024).

One way to enforce the contract is constrained decoding: you only let the model emit tokens that fit a grammar or a finite-state machine, so the output is valid JSON, a legal tool call, or a DSPy-shaped record. SGLang (Zheng et al., 2024) is the lecture’s example: it uses finite-state machines, among other systems tricks, to keep structured generation fast. DSPy signatures are a higher-level version of the same idea: you declare the fields you want, and the framework takes care of getting the model to fill them.

You can also constrain the model outside free text generation. A small classifier head on embeddings can map a state to a label. The lecture mentions a current slogan, “System One” models: roughly, classifiers with adaptable options, used for decisions rather than for open-ended prose (the lecture points at contrastive-lm.notion.site). And for tools, the ancestor paper is Toolformer (Schick et al., 2023): a language model that learns, from its own traces, when to call an API and how to put the result back into the text.

Context engineering

Prompt engineering was about the wording of a single request. Context engineering is about everything that goes into the window of an agent over a long run. The lecture’s picture is from Anthropic’s Effective context engineering for AI agents.

Figure 2: Prompt engineering for a single turn, versus curating what of a much larger possible context actually enters an agent’s window — CS 329Z Lecture 2 (Diyi Yang, Michael Ryan, John Yang), slide p. 161. Original: Anthropic, Effective context engineering for AI agents.

What might go in that window? The lecture’s list is short and complete enough to start from:

• the system prompt
• the message history
• memories
• tool-call results
• attached files

The temptation is to put all of it in, every time. That is usually a mistake.

Context rot, bloat, and the cache

Longer is not always better. Oolong (the lecture stylises the name as OOLONG) measures long-context reasoning and aggregation: as the context grows, scores fall, on both synthetic and more realistic splits. The lecture uses that plot as a warning labelled context rot.

Figure 3: Context rot: scores on Oolong drop as the context gets longer, rather than staying flat out to the advertised window — CS 329Z Lecture 2 (Diyi Yang, Michael Ryan, John Yang), slide p. 164. Original: Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities. The lecture writes the name as OOLONG; the paper’s title is Oolong.

The lecture’s practical advice, which students also meet in the course homework, is that there is no single right context. You experiment. A few concrete habits help:

• Avoid bloat. Sometimes a few tools are enough. The lecture points at Pi, which exposes just read, write, edit, and bash — a case made in The harness tax on arena.ai. Recovery of the rest can wait for retrieval, which is the subject of Why RAG and Indexing for RAG.
• Add sequentially. If you insert or edit tokens earlier in the context, the positional embeddings of everything after them change, and the KV cache for that suffix is invalidated. Append new material where you can, so prefill work is not thrown away. This is the inference engine leaking into the agent’s bookkeeping.
• Evaluate the choices empirically. Toolformer is the lecture’s reminder that you can set up tests and measure whether a context change helped, rather than arguing from taste.
• Compact carefully. By the time you compact, the context is already very long. Keep too much and you will have to compact again soon; drop too much and you lose what the task still needs.

Figure 4: Inserting or editing earlier tokens changes later positional embeddings and invalidates the KV cache for the suffix — CS 329Z Lecture 2 (Diyi Yang, Michael Ryan, John Yang), slide p. 169. Lecture original.

A different philosophy: recursive language models

Instead of stuffing a huge window, recursive language models treat the context as something the model can explore with its own sub-calls: look at a slice, recurse, and only bring back what is needed. The paper is Recursive Language Models (2025); the lecture also credits Alex Zhang’s write-up. On one Oolong setting in that write-up, an RLM built on a smaller model can outscore a larger model used in the ordinary way, and outscore a ReAct baseline. Those bar heights are the source’s numbers, not a rerun of our own.

Figure 5: Recursive LMs on one Oolong setting with a 263k-token context, as reported by the source — CS 329Z Lecture 2 (Diyi Yang, Michael Ryan, John Yang), slide p. 173. Original: Recursive Language Models (see also arXiv:2512.24601).

In one table

Habit	Why it matters
Structured I/O	Compound systems need a contract, not free-form prose between every part
Constrained decoding	Enforce the contract at sample time (FSMs, grammars, DSPy signatures)
Small decision models	Classifiers, or “System One” heads, for some choices instead of a long generation
Context engineering	Curate the window: system prompt, history, memories, tool results, files
Context rot	Longer contexts can hurt; measure, do not assume
Few tools, then retrieval	Avoid harness bloat; recover extras when needed
Append, do not reshuffle	Protect the KV cache
Compaction / recursion	When the window is full, summarise carefully — or let the model recurse into the text

FAQ

Is context engineering just a new name for prompt engineering? It is a wider job. Prompt engineering is usually a single wording. Context engineering is the running policy for what an agent can see: what you include, what you retrieve, what you drop, and when you compact.

Why not always use the model’s maximum context? Because the model may not use that length well. Oolong is the lecture’s exhibit: quality can fall as you fill the window, even inside the advertised limit.

Why does editing the system prompt mid-run get expensive? The prompt is a prefix. Change it, and every later position shifts, so the cache built for that suffix is no longer valid. You pay a fresh prefill. This is one reason to keep stable instructions stable, and to append results at the end.

What is coming next in the course? The lecture ends with “see you on Wednesday for RAG.” Retrieval is the systematic way to keep the window small and still reach a large store of files and memories. That lecture is now a set of pages starting at Why RAG.

See also

• What is a language model?
• Language model architecture
• Language model training
• Language model inference
• Compound AI systems, Agent architectures, and Key challenges in building agents
• Why RAG, Indexing for RAG, Retrievers for RAG, Agentic RAG, and Failure modes of RAG
• RAG and agents: Resources