How agents improve over time: model, harness, context

Key ideas

• An agent is three parts you can improve separately: the model, the harness that runs it, and the context it reads.
• Changing a model's weights is slow and costly, so a person starts it: supervised fine-tuning for narrow tasks, reinforcement learning for the core model.
• Harness changes reach every user, so even when an agent proposes them from traces, a person approves them.
• Context is plain text, so it's the cheapest part to change, and it can be kept per user or per organization as memory.
• Every improvement loop rests on traces, which show exactly what the agent did, and evals, which define what good looks like.
• The loop is: run the agent, get feedback, read the trace, save it as an eval, then change the weights, the code or the context.

Harrison Chase splits an agent into three parts you can improve: the model’s weights, the harness code that runs it, and the context it reads. Context is the cheapest to change and the model the most expensive, but all three run on the same two inputs: traces of what the agent did, and evals that say what good looks like. These are my notes on the talk. If you run an agent, the useful part is the loop he lays out for turning one bad trace into an eval and a fix, and his list of what makes evals hard to build.

Model, harness, context

The model does the reasoning and makes the decisions. Chase picks one as the core workhorse and notes that an agent often has other, smaller models around the edges, for guardrails and evaluation checks.

The harness is everything around the model. It decides what the model sees, carries out what the model decides to do, checks the results, and saves things to disk or a database. LangChain’s own harness is Deep Agents, an open source project that ships a file system, a sandbox to run commands in, skills, memory, summarization of long threads, human approval steps and subagents. Claude Code and Codex are harnesses in the same sense.

Context is the knowledge the agent reads. That’s mostly text, such as instructions, files and memory, plus the tools it can use.

Any of the three can improve, and Chase sees three levels of automation for doing it. An engineer can drive the change by hand, which is most of what AI engineers do today. A background process can propose changes for a person to approve. Or the whole thing can run on its own. He calls that last one the North Star and says it’s nowhere near ready.

Changing the model

Improving a model means changing its weights. An agent may use several models, so the first choice is which one: the core model, or a guardrail or eval model at the edge. Chase says a person almost always starts this, because training costs time and money and isn’t something to run continuously in the background.

There are two methods. Supervised fine-tuning takes traces, keeps the ones where the agent did well, and trains the model on them. He sees it used mostly for narrow tasks such as classification, extraction and summarization. LangChain’s tool for it is LangSmith Fine-Tuning, a command-line tool called smithtune, released in beta on September 24, 2026. (LangSmith is LangChain’s platform for tracing and testing agents.) It pulls trajectories from a LangSmith project, filters them, puts them in the training format and hands the job to one of two model-hosting companies: Fireworks, which runs the training as a managed service, or Baseten, through Loops, its training kit that’s in early access. For now it does supervised fine-tuning only.

For the core model on harder tasks, a few teams use reinforcement learning. Instead of examples, you give it tasks. The agent attempts each one, each attempt is scored, the model is updated from the scores, and it runs again. Chase calls it a lot more complicated with a higher ceiling, and in his view there are several managed services for supervised fine-tuning but no good one for reinforcement learning yet.

Changing the harness

The harness is code, prompts and tools, so changing it means editing those. The process is to read traces, find what went wrong and change the harness to fix it. A harness change reaches every user, so Chase wants a person reviewing each one, even as parts of the process get automated.

LangSmith Engine is LangChain’s attempt at automating the first part, launched May 13, 2026. LangChain calls it a proactive agent engineer. It watches production traces, groups failures into named issues, and proposes fixes: it opens pull requests, adds evaluators, and adds failing traces to eval datasets. Under the hood, Chase says, it’s Deep Agents running in a sandbox with the LangSmith command-line tool, plus some specialized subagents, skills and prompts.

Changing the context

Context is the easiest part to change because it’s text. That also means it can change at more levels than the core agent: per user or per organization, as memory that builds up over time. The process is the same as for the harness, reading traces and fixing what they show, but cheaper.

Chase points to two companies he thinks do this well:

• Hex, a data analytics product, sells a shared context layer that improves over time as part of the product. Its Review Agents analyze how people use its data agents, find gaps in the context and suggest fixes, through a feature Hex calls Context Suggestions.
• Profound, an AI marketing product, keeps context at two levels. Its Context Manager distills what it knows about a company into six Brand Records, and it collects Personal Records from each person’s interactions with its AI Marketer.

LangChain’s own experiment is OpenWiki, an open source command-line tool that writes and maintains documentation of a code repository for agents to read. It installs into coding agents including Claude Code and Codex, and a LangSmith connector pulls in traces so the docs reflect how the code behaves when it runs. Chase describes the aim as seeing what Codex or Claude Code struggle with and feeding that back into the wiki.

Traces

What the three loops share, in Chase’s view, is traces and evals. The hardest part of any loop is deciding what to remember, what to forget and how, and that’s prompting work specific to each domain. Traces and evals are the part a platform can help with.

LangSmith’s vocabulary nests: runs make up a trace, traces make up a thread, and a trajectory is a standard view of what happened. What matters is that each keeps the exact inputs, the exact outputs and the exact order of events. Whether you’re fine-tuning a model or having a coding agent debug the harness, that’s the record you need.

Chase argues traces are a new kind of data, because agent failures are mostly failures of meaning and logic. They show up in the conversation, not in ordinary logs, so finding and querying them needs different tools. That’s why LangChain built SmithDB, which it describes as a data layer built for agent observability and evaluation, announced May 13, 2026. The newest piece is AI Search, launched the day of the talk. You describe what you’re looking for in plain words. Ordinary filters run first, then a model checks each remaining trajectory against your description.

Evals

Chase likes the saying that evals are the new PRD, or product requirements document. An agent has so many edge cases that the clearest way to say what it should do, and what it shouldn’t, is a set of evals. They’re also what keeps you in bounds while you iterate.

He calls evals really hard to build, with no silver bullet, because both the test cases and the graders have to fit the domain. He breaks the difficulty into three parts:

• Realistic inputs. You often can’t predict what people will ask, including the requests that are off topic or nearly on topic. Production traces are how you find out.
• What good looks like. The simplest check is an exact match, such as a number or a phrase the answer must contain, and it breaks down for complex agents. The next step is criteria, positive or negative. They can judge the final answer, the end state of whatever the agent changed (a database, a set of files), or the exact sequence of tool calls. He thinks subject matter experts are best placed to write them.
• A place to run them. An agent that writes to production systems can’t be pointed at them for evals, so teams build simulation environments: a mock database or a dev environment the agent can change freely. That’s easier for systems you own than for something like Salesforce, and he says LangChain spends a lot of time on what these environments should look like.

Red teaming

There are a few ways to find an agent’s mistakes. Reading production traces only catches what has already happened, and something rare can do damage before you see it. Having domain experts try the agent works, but it’s manual and costly. Red teaming looks for problems ahead of time. LangChain added it to LangSmith Engine in private beta on September 24, 2026. Engine forms hypotheses about how the agent could fail, sends up to two synthetic requests to test each one, and judges the responses.

The hard part is correctness. Deciding whether an answer is right takes domain knowledge, and Chase points out that if you had a reliable judge, you’d just put it inside the agent’s loop. Since there isn’t one, LangChain probes the agent in ways that make the right answer knowable. His example is sending two questions that mean the same thing. If the answers mean different things, one of them is wrong, and you can work out which. That’s too expensive to do while the agent is working, but it gives a reference point. It’s a form of metamorphic testing, and Chase’s earlier open source tool for it, LangFuzz, is on his GitHub.

Correctness is one category. Others cover security and fairness, and the plan is a menu of checks that each produce a red-teaming report on demand. Red teaming runs the agent, so it needs the same simulation environments as evals, which LangChain’s labs team is researching.

The improvement loop

The flywheel Chase ends on has five steps:

1. Run the agent.
2. Get feedback on how it did, from a person or from an automated judge running on live traffic.
3. Open the trace to see what led to that feedback.
4. Add the case as an eval data point, so later changes are tested against it and earlier fixes don’t regress.
5. Propose a change to the model’s weights, the harness code or the context.

The future version is the agent running that loop on itself: noticing its own mistake, writing the eval, and proposing its own fix to any of the three parts. Chase calls that a North Star for Engine, with a lot of nuance still to work out.

How to try it

The talk is about LangChain’s tools, but the loop doesn’t depend on them. For one agent you run:

• Before fixing a failure, decide which of the three parts the fix belongs in. Context is the cheapest to change, so check it first.
• Keep full traces, with exact inputs, outputs and order, not just error logs.
• Each time the agent fails, save that case as an eval before you fix it, so the fix is tested and stays fixed.
• Write each eval’s criteria against what matters for that task: the answer, the end state of what the agent changed, or the tool calls it made.
• Give the agent a sandbox or a mock of anything it writes to, so evals never touch production.
• Keep a person approving harness changes, since they reach every user.

Key terms

Harness

The system around a model that decides what it sees, carries out the actions it chooses, checks the results and saves state. Deep Agents, Claude Code and Codex are harnesses.

Supervised fine-tuning

Training a model on examples of good runs so it learns to repeat them.

Reinforcement learning

Training by having the agent attempt tasks, scoring each attempt and updating the model from the scores.

Run

In LangSmith, one step: a single model call or tool call.

Trace

All the runs for one request, with their exact inputs, outputs and order.

Thread

The traces of one multi-turn conversation.

Trajectory

LangSmith's flat, ordered view of the messages along the path an agent took.

Eval

A test case and a way of grading it that together define what a good result looks like.

Red teaming

Probing an agent on purpose to find its failures before users run into them.

Metamorphic testing

Changing an input in a way that shouldn't change the answer, then checking that it didn't.

Quotes from the talk