Is An Illustrated Guide to AI Agents worth reading?
A review of Grootendorst and Alammar's O'Reilly book on AI agents (2026) for business readers: who it's for, the chapter that earns the price, what's already dated, and what to do with it this week.
Is An Illustrated Guide to AI Agents worth reading?
By Andrej Ruckij · October 28, 2026 · updated November 4, 2026
TL;DR: Yes, if you build agents or buy them. Maarten Grootendorst and Jay Alammar’s book (O’Reilly, September 2026) explains how an AI agent works through hundreds of diagrams, and you can follow it without writing code. Most chapters cover what practitioners already know. Chapter 7, on evaluating agents, is the one to pay for: it separates an agent that can succeed from one that succeeds every time, and shows how to measure the difference. Ignore the model names in the examples; newer models have replaced them.
Why I bought it
I build agent workflows for clients and for our own marketing, so I bought it. Four Claude readers took a quarter of the book each, and every claim they reported was checked against the text. It turned up four places where our own wiki was wrong. The one that matters most for buyers is below.
What is in An Illustrated Guide to AI Agents?
Ten chapters in two parts. Part I builds one agent from its parts: the language model, reasoning models, memory, tools (including MCP, the standard way to plug tools into a model), planning, and evaluation. Part II covers teams of agents, agents that see images and video, and coding agents. A small Python agent, the TinyAgent, grows chapter by chapter. The code lives in the book’s GitHub repository, which the authors say they keep updated as libraries change.
The diagrams carry the book. One drawing of an agent’s anatomy opens Chapter 1, and every later chapter zooms into one part of it. If you’ve ever tried to explain “memory” or “tool use” to a colleague and lost them, the figures do it in a page.
Who is the book for?
The authors write for several audiences at once: developers, people with an ML background, and “a technical leader, product builder, or curious reader” who wants the concepts without the code. That third reader is the one I’d send it to. A founder or marketing lead about to sign an agent contract gets the most out of it per hour.
If you already build agents, Chapters 2, 3 and 6 will feel familiar: how language models work, models that reason before answering, and step-by-step planning. Read Chapters 1, 4, 7 and 8.
pass@k vs pass^k: the idea worth the price
Chapter 7 separates two numbers that vendors blur. pass@k asks whether an agent succeeds at least once in k tries. pass^k asks whether it succeeds on all k tries. The first measures what the agent can do; the second, what you can rely on.
The book’s example comes from the SWE Atlas coding benchmark (Raghavendra et al., 2026). GPT-5.4 and Opus 4.7 both score 40 on a single try, which looks like a tie. When all three tries have to pass, they score 28 and 23. Given three tries, Opus 4.7 solves the most tasks at least once, at 60. Which model is better depends on which question you ask.
The benchmark is about code, and the book doesn’t extend it to business tasks. My read is that a business repeating the same task every week is asking the second question, and a demo answers the first. If a vendor can only show you a good run, ask how many of their runs look like it.
Other ideas from the book
Most jobs don’t need an agent. The book draws autonomy as a range: a single prompt, a fixed sequence of steps, or an agent that picks its own steps. Agents suit problems “where the goal might be clear, but the path to reach it is not.” If you can write the steps down, build a fixed workflow. Our wiki’s comparison page recommended agents for “defined, repeatable steps”, which had it backwards, and we’ve corrected it. The authors also argue that a system with many guardrails is often more effective than a fully autonomous one.
When one model grades another’s output, it tends to favour confident answers, whichever option it reads first, and its own model family. The book’s fixes are a judge from a different model family, swapping the order of what it compares, and checking the judge’s verdicts against human labels. I run an AI-tell checker on every article on this blog, this one included. Its log shows 37 of 38 checks passing, but it records only the final verdict, and the model that rewrote each draft also judged it. So the log can’t tell me whether the judge has ever caught anything on its own. For this article I had a separate model do the read, and it flagged an overstated sentence that the writing model had let through.
Approval prompts only protect people who can judge the action. Coding agents ask the developer to approve each command, which works because the developer understands the command. The book argues that safety has to be designed in when the person clicking “approve” can’t judge it: give the agent only the access it needs, and scope tasks so a mistake stays small. A 2026 study it cites (Your Agent, Their Asset, on the OpenClaw personal agent) found that poisoning any one part of an agent’s saved state (its skills, its identity, or its stored knowledge) raised the average attack success rate from 24.6% to 64–74%.
Chapter 4 explains agent memory plainly: what an agent keeps within one conversation, what it keeps across conversations, and how searching stored documents (RAG) works as an agent’s long-term memory. We rewrote our agent memory page from it.
What’s weak or already dated?
The models in the examples, such as GPT-5.4 and Opus 4.7, have newer successors as of October 2026. The authors say up front that they teach fundamentals instead of chasing releases, and the fundamentals hold. MCP changed again in a July 2026 spec update, and the book gives no spec version, so check the current spec before building on its MCP chapter.
One technical slip: the book describes MMR, a method for picking varied search results, as if a higher λ meant more diversity. In the standard formula (Carbonell and Goldstein, 1998) a higher λ puts more weight on relevance. It won’t matter if you read for concepts. If you implement it, check it.
Is it the best book to understand AI agents?
For a non-engineer who wants an accurate picture without code, I’d start here, mostly because of the figures. For an engineer it’s a good map with one excellent chapter on evaluation. If you want a PDF, buy it from O’Reilly; the official code is in the GitHub repo above.
What to do this week
This is for teams that buy, pilot or run agents. If you only use ChatGPT yourself to draft text, this section isn’t for you.
If you’re buying or piloting an agent:
- Pick one real task the agent will do, with a clear right answer (a report, a record update, a reply to a customer).
- Run that same task 5 times in fresh sessions.
- Count how many of the 5 are fully right. 5 of 5 is your pass^5. If it’s 3 of 5, it works only some of the time, so ask the vendor for repeated-run results on your own data before signing.
If you already run an agent:
- Write 5 to 10 test cases from real usage, including at least two that failed before. The authors put it plainly: “even a small custom eval of 5 or 10 cases can give eye-opening signals.”
- Rerun the set after every change to the prompt, model or tools, and keep the results in one sheet.
If you track how AI assistants mention your brand. This is my extension of the book’s idea, not something it tests. Ask ChatGPT the same buyer question (“best [your category] for [your customer]”) 5 times, each in a new temporary chat so earlier answers don’t carry over. Write down two numbers: how many answers name your brand at least once, and whether all 5 do. The second is the one to track month to month.
If you’re deciding whether you need an agent at all: write the steps down. If you can, build a fixed workflow and keep the agent for the part where the steps change every time.
Sources
- Grootendorst, M. and Alammar, J. (2026). An Illustrated Guide to AI Agents. O’Reilly Media, first edition, September 2026. Book page and errata
- An-Illustrated-Guide-To-AI-Agents (official code repository)
- Raghavendra, M. et al. (2026). SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution. arXiv:2605.08366
- Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw (2026). arXiv:2604.04759
- Carbonell, J. and Goldstein, J. (1998). The use of MMR, diversity-based reranking for reordering documents and producing summaries. SIGIR 1998
- How to evaluate AI products and agents (Primores wiki, updated after this book)
I post one short, practical take like this every day in the AI Daily for Business channel: t.me/aidaily_business