Language model architecture

From n-grams to Transformers, and why linear attention is in the mix

The shape of the typist

Source

A reader’s guide to part of CS 329Z Lecture 2 (Diyi Yang, Michael Ryan and John Yang; Engineering AI Agents, Stanford, autumn 2026). The outline and the pointers come from that lecture; the wording on this page is ours. The course is also listed under Resources.

Once you know that a language model is a next-token distribution, the next question is what computes it. The architecture is the typist’s memory: how much of the past they can see, how much that look-back costs, and what they have to throw away to go faster.

This page follows the lecture’s short history, then the Transformer, then linear attention and the hybrid designs that try to keep both.

Everyday example

Think of reading a long letter. Full attention is like being able to glance back at every earlier sentence whenever you need it: accurate, and slower as the letter grows. A recurrent model is like keeping only a short mental note of what has been said; cheap, and easy to forget a name from page one. Linear attention is a compressed notebook of constant size: you can keep writing for a long time without the notebook getting thicker, but the notes are a summary, not the original pages. A lot of current systems keep a few “glance back at everything” pages and use the notebook for the rest.

A brief history

The lecture’s sketch is short on purpose:

N-gram models (the standard toolkit well into the 2000s, and in use long before) guess the next word from a fixed window of previous words. They are simple, and they hit a wall: a slightly longer phrase, or a slightly new wording, and the table has never seen it.
The first widely cited neural language model is Bengio, Ducharme, Vincent and Jauvin, A Neural Probabilistic Language Model (JMLR, 2003). Words become vectors; a network shares parameters across contexts instead of storing a giant sparse table.
Recurrent neural language models, associated in the lecture with work around 2010 (Mikolov and colleagues), fold the past into a hidden state that is updated token by token.
Long short-term memory was introduced by Hochreiter and Schmidhuber in 1997. The lecture’s phrase is that it was popularised for language models in 2014 — a fair mid-2010s date for the wave of LSTM language models and sequence-to-sequence work, not the date the cell was invented.

Then the field changed shape.

Transformers

The leap from recurrence to self-attention is already told in two places on this wiki. The evolution of attention is the history: Bahdanau, Luong, then the 2017 decision to drop the recurrent backbone. Attention Is All You Need is the reader’s guide to that paper — encoder and decoder stacks, scaled dot-product attention, multi-head attention, masking, and positional encoding — with the original Figure 1. Transformer by hand works through one block with small integers.

What the lecture adds, for a builder, is the shape you actually meet today. The original paper is an encoder–decoder model for translation. Almost every chatbot you meet is a decoder-only Transformer: embeddings, positional information, blocks of masked self-attention and a feed-forward network, then a softmax over the vocabulary. The paper’s own comparison (Table 1, below) is why that look-back beat recurrence: a shorter path between distant tokens, and more work that can run in parallel, at quadratic cost in the length of the sequence.

Linear attention

Softmax attention is \(O(n^2)\) in the context length, and the KV cache grows with \(n\). That is a tax every agent pays when the transcript, the tools, and the files pile up. The quadratic bill, and the 2020 reformulation that attacks it, are in The evolution of attention: Katharopoulos, Vyas, Pappas and Fleuret rewrite attention with a kernel feature map so the cost can grow as \(O(n)\), with a constant-size state — a different computation, not a faster softmax. FlashAttention, on the same page, is a third idea: keep the original softmax maths, but run it with fewer trips to GPU memory.

The lecture’s comparison is the one to keep for builders:

flowchart TD
  X["Tokens in a long context"] --> S["Softmax attention"]
  X --> L["Linear attention"]
  S --> S1["Each new token can look at every earlier token"]
  S --> S2["Time and KV memory grow with n squared and n"]
  L --> L1["Each new token updates a compressed state"]
  L --> L2["Time is linear; memory can stay constant"]
  S1 --> H["Many systems use both"]
  L1 --> H

The gain is speed and memory. The cost, which the lecture shows with RULER (Hsieh et al., 2024), is long-context quality: a compressed state is a worse look-back than attending to the original tokens. A useful side note from the lecture: neither the Transformer nor linear attention forces a hard context limit by itself. What usually caps you is how the model was trained to use its positional embeddings. In practice you are limited to the context length the model learned.

Mixing the two

So people mix them. Some layers, or some heads, keep global softmax attention; the rest use a linear or similarly cheap recurrence. The lecture illustrates this with figures credited to Songlin Yang, DeltaNet Explained (Part III) — the same line of work that sits behind flash-linear-attention — and with a recent hybrid, Kimi K3 (Moonshot, 2026).

Why this matters for agents

The lecture closes the architecture section with a builder’s list, not a theorist’s:

Agents meet extremely long contexts.
Inference efficiency is a design choice, not a later optimisation.
Knowing the architecture helps you pick a model whose memory and attention pattern fit the job.
The harness around the model should play to its strengths and cover its weaknesses. That is the job of an agent engineer, and it is the subject of LLMs for agents.

In one table

Architecture	What it keeps of the past	Cost of a longer context
N-gram	A fixed window of previous tokens	Tables blow up; unseen phrases get no help
RNN / LSTM	A hidden state, updated token by token	Cheap per step; information has to survive the state
Softmax Transformer	Every earlier token, via the KV cache	Time \(O(n^2)\), memory \(O(n)\)
Linear attention	A compressed, often constant-size state	Time \(O(n)\); weaker long-context recall
Hybrid (e.g. mixed global + linear)	A few full look-backs, plus a cheap state	A compromise, now common in practice

FAQ

Why did Transformers replace LSTMs for language models? Training can be parallel across the sequence, and the path from a token far back is short. That tradeoff is Table 1 below, and it is unpacked in The evolution of attention and in the reader’s guide to the 2017 paper.

If linear attention is faster, why keep softmax attention at all? Because the cheap state is a summary. Benchmarks such as RULER show that long-context work still benefits from being able to look at the original tokens. Hybrids are an attempt to have both.

Is the context window a law of the architecture? Not in the strict sense the lecture is careful about. Attention does not forbid a longer sequence. What usually binds you is the positional scheme the model was trained with, plus the memory bill of the KV cache.

Where should I go for the attention equations? The lecture walks through queries, keys, values, the causal mask, and multi-head attention. We unpack the same maths in Attention Is All You Need. This page stays with the design choices that leak into agents.