flowchart TD Rel["Reliability: errors compound"] Train["Training: sparse reward, long horizon, costly rollouts"] Hor["Long horizon: context grows and drifts"] Saf["Safety: finishing the task is not the same as behaving well"] Ev["Evaluation: what to measure, and how"]
Key challenges in building agents
Reliability, training, evaluation, and safety
When the demo is not the system
Source
A reader’s guide to part of CS 329Z Lecture 1 (Diyi Yang, Michael Ryan and John Yang; Engineering AI Agents, Stanford, autumn 2026). The outline and the pointers come from that lecture; the wording on this page is ours. The course is also listed under Resources.
An agent that looks good in a one-shot demo can still fail as soon as it has to run for more than a few steps, on a slightly different task, in front of a person who did not write the prompt. CS 329Z Lecture 1 groups the open problems under five headings: reliability, training, long-horizon behaviour, safety, and evaluation. They overlap, but they are not the same complaint.
Everyday example
A single typo in a one-line email is an annoyance. The same typo in a ten-step travel booking — wrong date copied into the hotel, then into the invoice, then into the calendar — becomes a ruined trip. Agents live in that second world. Each step takes the last step as input, so a small mistake does not stay small. That is why reliability, evaluation, and safety have to be designed for trajectories, not only for single answers.
Reliability, and why evaluation has to measure it
Errors compound in an agent setup. A slightly wrong tool call produces a slightly wrong observation, which produces a worse thought, which produces a worse act. A system that is “usually fine” at each step can still be unreliable over a full task.
The lecture asks three evaluation questions that are easy to skip if you only report a success rate:
- Consistency across runs. If you give the agent the same task twice, does it reach the same outcome?
- Robustness to perturbations. If you make a small, realistic change to the prompt, the tools, or the environment, does the behaviour hold?
- Predictable failure. When it fails, is the failure legible and recoverable, or does it fail in a way you cannot anticipate?
These are reliability questions, and they are also evaluation questions. A leaderboard that only counts “task completed” will not show you any of them. The lecture points to the Holistic Agent Leaderboard and its reliability dashboard as a public attempt to make that gap visible — a capability–reliability gap: agents can look able, and still not be dependable.
Kapoor, Stroebl, Siegel, Nadgir and Narayanan, AI Agents That Matter (arXiv, 1 July 2024), make a complementary argument about current agent benchmarks: too much weight on accuracy, not enough on cost; model-developer and application-developer needs mixed together; weak or missing holdouts, so agents overfit the benchmark; and evaluation practice that is hard to reproduce. Their title is the point. We want agents that matter in use, not only agents that climb a board.
Training, and the long horizon
Training an agent is hard for reasons that reinforcement learning already knew about, and that language-model loops have not removed.
- Sparse reward. Feedback often arrives at the end of a long attempt, if it arrives at all. Most tokens and tool calls get no direct signal.
- Long horizon. The task takes many steps. Credit assignment — which thought actually helped? — becomes a guess.
- Expensive rollouts. Each attempt may mean many model calls, many tool calls, and a live environment. You cannot cheaply sample a million games in a simulator if the “game” is a cloud account or a colleague’s inbox.
Long-horizon behaviour is also a systems problem at inference time, not only a training one. The context grows. Early goals drift or get crowded out. The agent may forget the constraint you stated at the top, or spend its window on a raw log instead of a useful memory. That is why the architecture page spends time on episodic, semantic, and procedural memory: without some way to write down and retrieve what matters, a long run is just a short run with more tokens in the way.
Safety: finishing the task is not enough
The lecture’s slogan is easy to remember and easy to ignore in a demo: task success is not the same as safe behaviour. An agent can book the hotel, edit the repo, or send the email, and still have done something you would not have allowed.
A few concrete lines of evidence the lecture points to:
- HarnessAudit is a benchmark aimed at this gap: whether a harness complies with safety requirements, not only whether a task completes.
- Agents that share a computer with a person can lack a sense of when not to act, or of what the person is in the middle of. The lecture points to a hands-on write-up of an early computer-use agent (Platformer on OpenAI’s Operator) as a picture of missing collaboration awareness.
- Visual attacks can be almost boring, and still work. Attacking Vision-Language Computer Agents via Pop-ups (Zhang, Yu and Yang, 2024) shows that carefully designed pop-ups — the kind a person would usually recognise and ignore — can draw a vision-language agent into clicking them instead of finishing the task. In OSWorld and VisualWebArena the authors report an average attack success rate (the agent clicking the pop-ups) of 86%, and a 47% drop in task success. Simple defences such as “ignore pop-ups”, or an advertisement notice, did not fix it. (The lecture slide rounds the first of those figures; the numbers above are from the paper.)
- Privacy is a behaviour, not only a policy document. PrivacyLens (Shao, Li, Shi, Liu and Yang, NeurIPS Datasets and Benchmarks 2024) evaluates whether language models respect privacy norms in action, not only when asked a static question.
- Misalignment and sycophancy — telling you what you seem to want, or exploiting a reward signal — remain live issues once an agent can act. The lecture points to Anthropic’s write-up of reward tampering research.
- Multi-agent collusion is a newer shape of the same worry: several agents, or an agent and another system, can interact in ways no single-agent test would show. The lecture points to a METR investigation (26 August 2026) as a recent case to learn from, rather than as a folklore story.
None of these replaces a safety case for your own system. They are a reminder that “it completed the issue” is an incomplete report.
In one table
| Challenge | The short version | A place to look |
|---|---|---|
| Reliability | Errors compound; success rate is not dependability | HAL reliability dashboard |
| Training | Sparse reward, long horizon, expensive rollouts | The RL-era loop on the history page |
| Long horizon | Context grows and drifts | Memory types in Agent architectures |
| Safety | Task completion is not safe behaviour | HarnessAudit, PrivacyLens, Zhang et al., 2024 |
| Evaluation | What you measure, and how you hold it out | Kapoor et al., 2024 |
FAQ
Can I fix reliability just by using a stronger model? A stronger core can help, and it can also fail faster, or more confidently, in the same loop. Reliability here is a property of the system: tools, validation, memory, retries, and how you measure them. That is why the compound systems page insists on the three layers — system, data, and evals — rather than only on the model.
Why is evaluation listed separately from reliability? Because you can evaluate the wrong thing very carefully. A benchmark that rewards a costly, fragile agent for a slightly higher success rate will push the field in the wrong direction. Kapoor et al. argue for looking at cost together with accuracy, for keeping model evaluation and application evaluation distinct, and for holdouts that make overfitting visible.
Is “the agent got the task right” a safety result? No. That is the lecture’s point, and it is why HarnessAudit, PrivacyLens, and the pop-up attacks are on this page together. You need checks for the behaviour you didn’t want, including on runs that look like successes.
What should I read alongside the lecture? Kapoor et al., AI Agents That Matter; Weng, LLM Powered Autonomous Agents, for the architecture these problems sit on; and Zaharia et al., The Shift from Models to Compound AI Systems, for why the unit of work is the system.