Chip Huyen's AI Engineering, Applied: Grading My Production Systems Against the Book

I just finished AI Engineering by Chip Huyen. It’s the clearest book I’ve read on building software with foundation models, and it pairs with my Hormozi book-notes posts (like $100M Offers Applied): those were about the business around the work, this one is about the work itself.

My situation: I run several production systems that lean on large language models. An autonomous operator that watches all my projects and proposes fixes as reviewed pull requests (Command). An agent that runs a live Etsy shop end to end (ShopForge). A demand engine that produces category-authority research and just put its first vertical live (DemandForge). And a set of global Claude Code skills (transcript-analysis, global-review-loop, and pattern-retrospective) that mine my Claude transcripts for the corrections I keep making, turn them into rules, and check whether those rules actually held.

Below is every durable idea in the book, ranked by how directly it lands on what I’m actually running. I graded my own systems on a does/partial/absent scale against each one, and I tried hard not to hand myself credit I haven’t earned. Each section ends with an honest status and a next step.

The framework in 30 seconds

Huyen’s spine is a progression: understand the model, evaluate it, adapt it with the cheapest lever that clears your quality bar, treat data as a first-class lever, then ship it economically and close the feedback loop. Underneath all of it, model selection and cost against quality show up at every layer. Her own framing is that fundamentals outlast tools, so the book teaches durable methods over this week’s library.

The core of the “adapt it” step is a ladder of increasing cost:

Her rule is to use the cheapest lever that clears your bar, chosen by the failure you actually see, not by default. I ranked the sections below by how directly each idea lands on systems I’m running today, strengths first.

1. Evaluation is the spine, and it’s the first thing I build

The book spends two chapters here, because open-ended output is far harder to grade than a classifier’s accuracy. The advice is to combine methods (exact and functional correctness, similarity, an AI judge, pairwise preference) into an application-specific pipeline. Huyen’s line is that not having a reliable evaluation pipeline is one of the biggest blockers to shipping.

Before Command runs a self-improvement experiment, it fixes the success criterion first (the metric, the direction, the minimum effect, one attempt) and can’t move it after seeing the data. And every grader it trusts to call a result proven has to show a near-zero false-positive rate on synthetic data where nothing changed. ShopForge runs three-tier quality assurance (technical checks, a seven-dimension visual score, and buyer-persona sub-agents) behind gates, with a hard rule that no agent grades its own work. My global-review-loop skill adversarially self-validates every rule it proposes before it ships, so a fix never lands on its own say-so. DemandForge is the youngest of these, and bringing its research output under the same discipline is the next build.

Status: Already happening across the established systems. Next step: stand up the evaluation harness on DemandForge’s research output before its live vertical scales.

2. An agent is a planner plus tools plus memory, and compounding error is why I gate everything

Chapter 6 defines an agent as an environment, a set of tools, and a planner. Tools fall into three buckets: knowledge augmentation (retrievers, APIs), capability extension (a calculator, a code interpreter), and write actions (sending an email, writing to a database). Planning is treated as generate, validate, execute, reflect. Her warning: the more automated the agent, the more catastrophic its failures.

The math behind that warning is just arithmetic. If each step is 95% reliable, ten steps in a row land around 60% (0.95 to the tenth is about 0.60). That decay is the entire case for planning, reflection, and hard gates.

I’m positioned here. ShopForge splits planner, creator, and verifier and runs sub-agents in parallel, with a decision gate on every action. Command puts four hard walls (spend, message, publish, irreversible loss) directly in the tool-call path as deterministic checks, so the model proposes and the harness only executes what passes.

Status: Already built across systems. Next step: adopt the agent-evaluation metrics Huyen describes but I don’t formally track in Command: valid-plan rate, tool-call accuracy per category, and steps and cost against a human baseline.

3. Prompt engineering is human-AI communication, and two of my products are authored context

Chapter 5 treats prompting as the cheapest lever, the one to exhaust before RAG or finetuning, and adds a section on defensive prompting against injection once tools are in play.

Two of my systems are, quite literally, authored context. Command’s whole product is instruction: it proposes CLAUDE.md rules, skills, and hooks as reviewable pull requests, so it writes config rather than just consuming it. My global skills mine the corrections I keep making and emit concrete rule and hook patch text. ShopForge keeps a prompt and template tier in its memory and lints prompts against a denylist.

The book’s sharpest addition here is the defensive half. My agents can send and publish, so those write-action tools carry a real prompt-injection surface, and hardening it is now on the tracked list.

Status: Happening, keep going. Next step: keep growing the open-source skills library, and harden the write-action tools against prompt injection.

4. Model selection and cost against quality is a per-task decision I already make

The book frames model choice as building your own private leaderboard for your use case, since public benchmarks weed out bad models but won’t find your best one. Cost against quality recurs at every layer: build versus buy, host your own versus an API.

I make this call per task. Command keeps the model out of its cheap watching loop entirely and only spends on the reasoning session. ShopForge sends image generation to one provider and reasoning to another.

The honest qualifier: that’s deliberate routing by task shape and cost, not Huyen’s eval-driven private leaderboard. I pick models by judgment, not by a measured head-to-head.

Status: Positioned here, but by judgment rather than measurement. Next step: build a small private leaderboard for the one model-choice decision I still make on feel.

5. RAG: I’ve built the textbook version, and my production systems use structured memory on purpose

Chapter 6’s other half is retrieval: retrieve, then generate, where retriever quality makes or breaks the result. A useful reader shorthand for the whole argument is that RAG is for facts and finetuning is for form.

I’ve built the textbook version twice. One is my own experiment: an image-search service running vector similarity search on Redis with OpenAI embeddings, seeded with synthetic data, plus an evaluation script that grades the pipeline with RAGAS-style metrics (faithfulness and relevance, the standard RAG scorecard). The other is client work on Akaya, a document-question-answering app doing production document RAG with embedding retrieval.

My production systems make a different choice, and it’s deliberate. ShopForge and Command assemble context through structured memory: a database lookup that pulls exactly the records a task needs. When the data already has structure, an exact lookup beats a similarity search, and embeddings would add fuzziness precisely where I want precision. For fresh research I skip both and fan out web searches, then adversarially verify what comes back.

The book still sharpened one real decision. DemandForge’s research layer is where retrieval quality matters most in my stack, so evaluating it, and deciding whether vector retrieval earns a place there, is now a tracked goal rather than a vague intention.

Status: Built where it fits, skipped where it doesn’t. Next step: evaluate DemandForge’s research retrieval and decide where vector search earns a place across my systems.

6. Dataset engineering: high-signal beats high-volume, and that’s how I record

Chapter 8 sets three co-equal criteria for a dataset (quality, coverage, quantity) and stresses diversity throughout, then lands on a durable line: a small amount of high-quality data can outperform a large amount of noisy data.

I already record this way. ShopForge keeps every error it makes, but it only keeps a success when the quality score attached to it has something to teach, so its memory stays small and useful. Command keeps a short metric history per project, and it also generates data where nothing changed on purpose, which it uses to prove its graders don’t claim wins on noise. My global skills read both of my Claude transcript archives but only act on the corrections that keep showing up, not every one-off. It’s the same instinct everywhere: a few records that mean something beat a pile that doesn’t.

Status: Happening. Next step: add an explicit diversity check to what ShopForge learns from, since the book argues diversity is a bigger lever than raw volume.

7. The data flywheel: the first rung is proven, the rest are pre-registered

Chapter 10 is about the system around the model: guardrails, routing, caching, observability, and the user-feedback flywheel as the durable moat. Its thesis is that many AI challenges are, at their core, system problems.

This is where my real work is right now. Command is proving its headline claim, that fixes accumulate across the fleet, up an explicit three-rung ladder: capability, generality, fleet value, with each rung sealed as a pre-registered experiment before the data comes in. The first rung is proven: a sealed 18-run controlled eval (two runs excluded by a pre-registered rule) showed every scored treatment session adopting a rule Command shipped while zero control or placebo sessions did, a one-sided Fisher exact p of 0.002 (from Command’s committed eval records; its repo is private, so cited rather than linked). I’m careful not to oversell that: rung one proves the delivery mechanism changes real agent behavior, and the fleet-value rung is the claim itself, still ahead. PlumageDispatch, DemandForge’s first live vertical, is out collecting real outreach replies right now, the first real-world signal any of my venture engines has produced. ShopForge’s three nested improvement loops, from per-task retries up to strategic review, are built and waiting on the signal only sales can provide. And my global skills close their loop by construction: they only act on patterns that actually recur in my transcripts.

Status: One rung proven under a sealed experiment; two rungs and two real-world signals in flight. Next step: drive Command’s generality and fleet-value rungs to their pre-registered verdicts, and read PlumageDispatch’s first replies as the venture side of the same loop.

8. Inference optimization: I solve it at the architecture level

Chapter 9 treats latency, throughput, and cost as first-class engineering, splits latency into time-to-first-token and time-per-output-token, and covers quantization, caching, and batching.

The biggest cost lever in my stack is architectural, and I’ve already pulled it: Command’s watcher is deterministic code with no model calls, so sensing the whole fleet costs no tokens at all, and spend only starts when a reasoning session actually acts. The deepest layer Huyen covers, token-level tuning of self-hosted inference, doesn’t apply to me by choice, since I run on hosted APIs rather than my own weights. What’s left is instrumentation: ShopForge doesn’t track its spend yet, which its own case study flags, and that’s the next add.

Status: Solved at the architecture level, instrumentation next. Next step: add cost tracking to ShopForge.

9. Models are probabilistic by construction, so I design around it

Chapter 2 makes the point that a model’s output is probabilistic as a direct consequence of sampling, so you engineer around inconsistency and hallucination rather than expecting determinism.

This one isn’t a to-do. It’s the assumption under everything above. ShopForge’s gates and Command’s deterministic walls exist because the model is a probabilistic component, not a deterministic one.

Status: Internalized.

10. Finetuning: deliberately absent, and I think that’s right

Chapter 7 treats finetuning as the most involved lever, to reach for only when prompting and RAG fall short, and makes the point that getting good data is the hard part, not the training loop.

ShopForge’s stated goal is to get measurably better over time without ever touching model weights, improving through memory and rules instead. That’s the one rung of the ladder I skip on purpose. The book leaves me comfortable with the choice: finetuning is the expensive option, and none of my systems have hit the failure that would justify it.

Status: Absent on purpose. Next step: none, unless a specific failure mode starts demanding weights.

The RAG and agent chapters, against current Anthropic guidance

I wanted to hold the book’s tool-use chapters up against how Anthropic frames the same ground today, so here’s that cross-reference. Chapter 6 is where the book most invites the comparison. These are my cross-references; neither the book nor Anthropic cites the other.

Huyen’s planner-plus-tools-plus-memory model lines up with Anthropic’s Building Effective AI Agents and its distinction between fixed workflows and open-ended agents. Her compounding-error argument is the empirical case behind their advice to use the simplest thing that works.

Her tool taxonomy maps onto Anthropic’s Writing effective tools for AI agents, which argues, as she does, that tool definitions deserve real prompt-engineering care. This is the part I’ve built against directly. MCP API Bridge is my open-source starter kit for wrapping any REST API as a typed, validated MCP tool, with the annotations and input schemas the assistant needs to call it safely. And I have an open pull request to the official MCP Python SDK that threads progress reporting through the server-session calls, so long-running tool calls can report back. It’s still open and awaiting maintainer review, not merged, but it’s the kind of plumbing this chapter is about.

Where they agree, the pairing is tight: her RAG section and Anthropic’s earlier Contextual Retrieval work attack the same bottleneck she names, retriever quality.

Here’s where they genuinely pull apart, in my reading. The book’s default answer once a model needs facts is the classic RAG pipeline: embed the corpus up front, then retrieve by similarity. Anthropic’s newer Effective context engineering guidance argues the opposite default for agents: keep lightweight identifiers (file paths, stored queries, links) and let the agent load context just in time through its tools, trading some speed for focus and freshness. That’s a different architecture, not a nuance, and my systems landed on Anthropic’s side of the line before I could name it: the structured-memory retrieval in section 5 is exactly the just-in-time pattern.

The second real gap is standardization. Huyen’s tool categories are abstract by design, and what actually happened since the book shipped is the Model Context Protocol turning them into concrete, shared interfaces. The book couldn’t cover it, and it’s the layer I spend the most time in.

Status: The book supplies the fundamentals; the two real deltas since it shipped are just-in-time context over pre-built pipelines, and MCP as the standard tool layer. I’m already built on both.

The plan, sequenced

Now

Next

These aren’t decorative. Each one is going into Command, my operator, as a tracked goal, and Command publishes its goal set to my goals page. So whether I actually did them is checkable, which is the whole point of the book’s final chapter.

The most important shift the book left me with isn’t a technique at all. It’s the confirmation that the loop is the product. Command’s first rung is proven under a sealed experiment, PlumageDispatch is collecting the first real market signal any of my venture engines has seen, and every next step above is an evaluation or feedback step. The flywheel only counts once it’s turning on real feedback, and mine is starting to turn.