Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

Loop engineering taught me two things about measurement the hard way: a run can reach the right answer by the wrong path, and hop count is not a measure of work, because the model batches calls. I graded two runs by reading the traces myself. That does not scale past two. What does it take to score the path an agent took automatically, and which of the things I noticed by eye can a program actually detect?

What I built

A scorer that reads an agent’s run records and reports ten properties of the path it took, and a labelled corpus to measure it against. The agent is the one from loop engineering: same fleet, same three tools, same question, plus a FINAL line so a program can read the answer.

Each property is defined twice, once as I would judge it reading the trace and once as code. Most are plain trace arithmetic: turns versus calls, repeated calls, errors and whether they were retried. The two that matter most are whether the evidence actually proves the answer, and from which turn it could have. Those needed something else: an oracle that knows what the tools mean. A top-ten list caps everything it didn’t return. A total that matches the devices already identified closes a range. From those two rules it bounds the rate every device could have, seen or unseen, and says whether anything could still beat the named answer. Replayed on loop engineering’s first trace, it sharpened my old by-eye claim: four of the six band queries could each be dropped on their own, but only three together.

The corpus is 24 runs chosen to take different roads:

• Opus 5 at low, medium and high effort, parallel and serial.
• Three runs on Haiku 4.5 in the normal environment, and two more in sabotaged ones, because Opus rarely goes wrong on its own.
• Sabotaged environments: the answer device deleted from the world, a tool that fails about 30% of the time, a tool description that lies about units, and a loop capped at three turns.

I froze the scorer before the corpus existed, and labelled every run by hand before running the scorer on it. The git history shows the order. A small LLM judge was written for the properties code can’t see, but it never ran: the shared API key ran out of credit the moment it started. Total spend: $1.96.

What surprised me

Final-answer accuracy hid most of what happened. 16 of 24 runs got the right answer for the world they ran in, and only 6 did enough to prove it. In the unmodified environment Opus was right 10 times out of 10 and proved it 3 times. All five low-effort runs were right and unproven; I had predicted at least two. Most of the gaps are absurd: a 2-minute device whose hours were never checked, and which would need under 3 hours to matter. But not all. One run wrote that it would double-check a device with 90 minutes of downtime, then didn’t. Had that device logged only 100 hours, it would have been the answer. I read the pilot trace myself and didn’t notice it was unproven.

The property I expected to be hardest scored perfectly. I predicted at least 85% agreement on whether a path was justified, and at most 60% on the exact turn it became so, with me as the lenient one. Both came out 24 of 24. In every run that was right but unproven, the oracle named the same missing device I had. The catch is that I made those labels by doing the bound arithmetic by hand, which is the oracle’s method. This shows code can do careful reading. It doesn’t show that careful reading is what a quick look gives you.

The number check was a fabrication detector in a corpus with no fabrication. On the dev runs, my first version accepted 85 to 93% of all integers under 1000 as following from the results. The typed rewrite agreed with me on 16 of 23 runs, and every run it flagged was flagged for a model name or firmware version: “InfusaLine 700” contains a 700. The real errors were a count written as a word and, three times, “314 minutes per 100 hours”. 314 really is in a tool result; the unit is wrong. Fixing the model-name bug lifts agreement to 20 of 23 and catches none of the four for the right reason.

The waste I came looking for wasn’t there. I predicted the median run would make three or more calls after the answer was already settled. 18 of 24 never settled it at all, and the other six made 0 or 1. Not one of the 324 calls was redundant: all 24 identical repeats were retries of a call that had just failed. On this task the failure is stopping early, not working too much.

The reasoning I most wanted to catch lives in a number. The pilot filtered on max hours 127, which is 88 minutes divided by the leading rate at that point, but its thinking never states the bound. I added a rule for derived filter values. On the corpus it fired twice, both times on runs keywords had already caught, because the other runs that used the bound rounded 127 up to 130.

What I’d do differently

Have someone else label, or label before building the oracle. Agreement between two things one person designed measures consistency. The number worth having is how often a quick human read disagrees with the proof, and I didn’t collect it.

Seed the failure each check exists for. Nothing in the corpus fabricated a number, so the number check was never tested against the thing it is for. The Haiku arm gave me natural wrong paths, but in the normal environment it was the same three-call shortcut every time.

Run the expensive comparison first. The judge was the last step, and the credit ran out before it. The half of the question it was for, what code can’t detect, has no measurement. It’s the same lesson as last time: run the risky thing on day one.

Measure under-work. I ported loop engineering’s measures of over-work: repeats, speculative calls, calls after the answer. The useful number was “right but unproven”, and it needed a model of these specific tools. The reliable half of this scorer is the half that knows the task, which means it won’t transfer.

The question

Every personal site I’ve seen is a list of finished things. That shape hides the part I actually care about: what I don’t know yet. Can the site itself be the instrument, so that looking at it tells me where my gaps are instead of just advertising what’s already done?

What I built

A map instead of a feed. Every topic is a node in a semantic space: dashed outline for things I want to understand, pulsing for things I’m building, solid for things I’ve finished and written up.

Positions come from embeddings. Each entry is embedded with Workers AI, projected from 768 dimensions to 2 with PCA, and laid out by meaning rather than by date. Related work ends up near related work without me arranging anything. Adding an entry is one markdown file; the map rearranges itself.

The whole thing is static. Nothing runs at request time, so the entire site is a folder of HTML on Cloudflare Pages.

Around that sits the machinery that keeps the instrument honest. Every push is type-checked, which matters because entries are typed data and a typo in a state should fail a build, not quietly drop a node. CI then re-embeds every entry and commits the new positions back if the map moved, so a diff in the positions file only ever means the writing changed. Starting work on a topic is one command that creates the repo, links it to the node and flips its state, and refuses any name that isn’t already on the map, so nothing unrelated on my machine can end up on GitHub by accident. Every writeup has the same four sections, and every project keeps a local notes file where surprises get written down the moment they happen, because the section about what surprised me is written weeks after the surprises.

What surprised me

The layout is deterministic across machines. I hoped a CI run on Linux would reproduce my laptop exactly, and didn’t expect it to. It did, byte for byte. That one property is what makes committing the positions file worth doing: any change in it is signal, not floating-point noise.

The geometry beat my argument. I predicted Graph RAG and knowledge graphs as memory would land together and probably want merging. They landed at opposite ends of the map: one reads as a retrieval technique, the other as a storage problem. I would have merged them on instinct. The map was right and I was wrong.

Prose moves the map. Metadata doesn’t. Writing the first section of one entry, about seventy words, was enough to move the map. Adding a repo link moved nothing, because links aren’t embedded. The map responds to what I say about a topic, not to its bookkeeping, which is what I wanted and still surprised me by how little text it took. It turned out to be too sensitive, which is why it’s no longer true: see the last point below.

The map once looked like it rearranged completely, and it hadn’t. Finishing the loop engineering writeup moved all twenty nodes, most by nearly the full width of the map. The structure hadn’t changed at all: distances between nodes were preserved and the whole thing had mirrored vertically, because a principal component is only defined up to its sign. “Positions changed” and “structure changed” are different claims, and I had written documentation that treated them as the same. Pinning the sign made that diff go to zero.

Real use found bugs that my testing didn’t. The mobile index shipped completely unscrollable, and passed my check because scrolling in code still works when scrolling with a finger doesn’t. The command that starts a project flipped a node to in progress without giving it anything to render. The site’s own entry linked to a GitHub account that doesn’t exist, because I typed it by hand before I knew the name; the script that derives the account has been right the whole time. And the CI type check caught a real bug on its very first run.

What I’d do differently

Use the instrument before perfecting it. When the first node was finally reported, the pipeline around the map was far more developed than the map itself: one finished topic out of twenty, and a CI system that re-embeds on every push. Building the apparatus was more fun than filling it, which is the same hazard loop engineering ran into on a smaller scale.

Distrust the early map. For weeks, almost every entry was a title and a one-line summary, and the layout was mostly an arrangement of those one-liners. Pairs like context compaction and LLM-as-judge sat almost on top of each other for no reason I could find. I’d either embed only entries that have real writing, or mark the early layout as provisional instead of drawing it with the same confidence as the later one.

Test with the input people actually use. Every bug that shipped had passed a check that tested a proxy for the real thing: programmatic scrolling instead of a finger, a script’s output instead of the rendered page, a link I typed instead of one I derived.

Keep the states honest. At one point seven nodes were pulsing at once. The point of the executing state is to show what I’m working on right now, and seven at once showed what I had started.

Place topics by what they are, not by what I wrote about them. Once eight writeups of about a thousand words landed, the map stopped arranging topics and started arranging word counts. Every finished node crowded onto one side and every unstarted one onto the other, and six reports about the same device fleet piled on top of each other. Positions now come from each topic’s title and one-line summary only, which puts every node on equal footing whatever its state, and keeps a node still when its report changes.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

Loop engineering ended on a hunch I never tested: the loop changes that actually break an agent are the ones that destroy information, not the ones that merely rearrange it. Compaction is information destruction on purpose. Every long-running agent eventually throws part of its history away and hopes it chose well. What can an agent lose and still finish, and can I find the point where compaction stops being housekeeping and becomes the cause of the failure?

What I built

The same rig as loop engineering, pointed at a different variable: the same 400 devices, three tools, ten-row cap and question. That task peaks around 6k tokens and never gets near a context limit, so nothing forces compaction on it. I forced it. Before each request, a transform rewrites the copy of the history that gets sent, while the loop keeps its own full record, so every run shows exactly what the model was shown and what it was denied.

The transform has three dials, moved one at a time: how many recent turns of tool results survive (three, two or one), what a cleared result turns into (a placeholder, the device ids it listed, or a digest of ids and numbers), and whether the model’s own earlier thinking goes too. Alongside it I ran the API’s own tool-result clearing, with its trigger dropped to 2,000 tokens so it would bite on something this small, keeping the last three or ten tool uses.

Each run is graded and counted for turns, tool calls, exact re-queries and cost. Halfway through I added one more measure, after one condition passed only because it never fired: the most results actually hidden from the model on any single request.

All of it ran on Claude Opus 5 at low effort, the setting with the least slack that still solves the untouched task. Ten conditions, three runs each (four for one), 31 runs, $7.60. The small n is the main limitation.

What surprised me

It never once gave a wrong answer. I predicted the heaviest condition would produce at least one confident wrong answer from a partial picture, most likely the plausible runner-up. In 31 runs, no answer was wrong. Every failure was a run that never answered at all. Compaction didn’t make the agent guess. It made it unable to finish.

Clearing results alone broke it, completely. I predicted that keeping only the latest turn of results would be survivable, because the numbers would already live in the model’s thinking. It went 0/3, and the three traces were almost identical. By turn three it had found the right device. Then, for the rest of its twenty turns, it alternated between fetching that device’s record and fetching the risk ranking, one call per turn, each fetch evicting the other: nine of one and eight of the other, in all three runs. The thinking I was counting on barely existed. At low effort the model’s whole output, thinking included, averaged about 140 tokens a turn.

The break is a cliff, and the task decides where it is. Keeping three turns: 3/3. Two: 3/3. One: 0/3. The answer needs two facts from two calls, and the second call depends on the first. Two turns can hold both. One can’t. No token budget explains that edge. The shape of the question does.

The ids were worth more than the numbers. I predicted that leaving device ids in a cleared result would change nothing. It went 3/4, against 0/3 for plain clearing. A cleared ranking that still names the top device lets the model fetch both records in one turn, which turns two dependent steps into two independent ones. The fourth run knew both ids and ping-ponged between them anyway, so ids make batching possible, not certain. And batching wasn’t enough on its own either: one run with its earlier thinking stripped fetched both facts together three times and still ran into the cost cap.

Removing more wasn’t worse. I predicted that losing results and losing thinking would only break it together. Losing thinking alone changed nothing, and losing both (1/3) was no worse than clearing results alone.

The API’s own clearing was harsher than mine. It keeps the last few tool uses, not turns, so part of a parallel batch is cleared before the model has read it. Keeping three: 1/3, and each failure ran to the cost cap. Keeping ten: 3/3.

The bill was right for the wrong reason. Every compaction condition cost more than doing nothing, as I predicted, but not because of context size. Each edit moves the prompt-cache prefix, so the history is rewritten to cache on every turn: about 67,000 cache-write tokens per run for one-turn clearing, against 3,700 for control.

What I’d do differently

Decide where to cut runs off before running them, and check what that costs. Failing runs ran into a limit at 48 to 65 cents each, while passing runs averaged 13. Most of the budget went on watching the same livelock play out again, which left three runs a condition, where 1/3 and 2/3 are the same result. The obvious fix, a 25 cent cap, would have bought about twenty more runs. It would also have killed the only successful run in two conditions, because both of those finished late. Where you cut a run off is a claim about which slow successes don’t count, and I’d want to make that claim on purpose.

Use a task with more than one dependency. One question with one pair of linked facts gives one cliff in one place. To know whether “the window has to hold everything the next step depends on” is a rule rather than an anecdote, I need tasks where that number is three, or five.

Measure what was destroyed from the first run. For part of this I believed a setting worked when it had simply never fired.

Test what I skipped. Nobody told the model its results would disappear. The obvious next condition is telling it, and seeing whether it batches, takes notes, or neither. The one prediction I never tested is whether high effort, with real thinking to carry things, survives what low effort can’t.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

Every agent I’ve built has run on someone else’s harness: a framework’s tool definitions, its context handling, its error behaviour. I’ve never built the surrounding machinery myself, so I don’t actually know which parts are hard and which only look hard from the outside. What does it take to build one, and what decisions does it force that I’ve never had to make?

What I built

A harness over this site’s own content, built from nothing, with each part in its own file and the decision it forced written at the top of that file. The model and the loop are deliberately plain, because loop engineering already studied loops. Everything else is the harness.

It sees a frozen snapshot of the site rather than the live one, so every run sees the same world. There are four tools that read (list, read a topic or one section of it, keyword search, and nearest neighbours on the map) and one that writes, which can’t touch the site. It only produces a proposal file for review, and only if a write policy outside the model allows it. Arguments are checked against the same schema the model was given, because the API doesn’t enforce that on its own. Every failure, whether a bad argument, an unknown topic, a refused write or a spent budget, comes back in one shape: what went wrong and what to do next. Results over a size cap are cut, with a note saying how to ask for less. The system prompt can optionally carry an index of every topic. Every run is written to a trace.

Most of the decisions I’d never had to make showed up before the model was called: snapshot or live data, whether writes exist at all, who validates arguments, what a failure looks like, how big a result may be, what goes in the prompt up front, and where the budget lives. Then I measured some of them. There were five questions with answers computed from the snapshot, under configurations that each changed one thing: the index preloaded, the cap removed, the cap cut to 500 characters, writes refused, and the site’s style rule either written into a tool description or enforced when the tool is called. That came to 49 runs on Opus 5 at low effort, for $1.40.

What surprised me

The comfortable version measured nothing. Base, preloaded, uncapped and read-only: 33 of 33 runs passed. On these tasks the configurations differed only in path and cost, never in whether the answer was right. Loop engineering warned me about exactly this and I built it anyway.

One constant was the dial again, and another part cancelled it. A 2,000 character cap turned one question from one call into seven, because the capped list forced the model to page through the clusters one by one. With the index preloaded, the same cap cost one call. The parts of a harness compensate for each other, which you only see by varying them together.

Search turned into a keyhole. At 500 characters, one run hit the cap eight times, then narrowed a search down to the exact phrase the answer sat in and read it through the search snippet, a 200-character window I’d built as a convenience. The cap limited reads and didn’t limit search. Every tool is a read path, whether I designed it as one or not.

Both failures were honest, and my grader couldn’t tell. The tight cap produced two failures. In one, the model said it couldn’t find the number rather than guess. In the other, it ran into the 25-call budget one topic short, and its reply named the topic it hadn’t been able to confirm and said why. My grader reads only the final line, so it scored that honest, partial answer the same as a confidently wrong one. I read the grade before I read the reply and wrote down the wrong conclusion. Grading the last line isn’t grading the run.

The model follows rules it’s shown, and I had never shown it this one. The site forbids em dashes. That rule lives in a file the harness never reads, and 9 of 14 proposals used one. Written into the tool’s description, 0 of 3 did. Enforced when the tool was called, the one that broke it was rejected and rewritten clean on the next try. The refusals worked the same way: when writes were off, every run tried once, took the refusal’s hint, and put the proposal in its reply instead.

What I’d do differently

Start with the version that can fail. The tight cap was an afterthought, and it was the only configuration that told me anything about correctness. I’d find the settings that break the task first, then compare designs there.

Audit every tool for what it can reveal. I thought of the cap as a limit on what the model could see. It was a limit on one tool. A search snippet, an error message that quotes its input, a neighbour list: each one is a way to see something, and the harness is only as tight as the loosest of them.

Carry the house rules in from day one, and enforce the ones that matter. A rule in a description worked here, but a description is advice and an enforced check is a guarantee, and the guarantee cost one extra call in one run out of three.

Read the runs, not the summaries. Twice I wrote a finding down from a summary instead of the runs. I recorded that every proposal had broken the style rule after looking at two of them; the real figure was five of the first six, and nine of fourteen overall. I recorded that a failed run had confidently returned an incomplete list; its reply said plainly what it was missing. Both came close to ending up on this page. Two runs per cell also can’t rank the configurations on cost, since the first run of each pays for the prompt cache and costs about half again as much as the second.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

The claim I keep repeating is that most agent failures are loop failures, not model failures. I don’t actually know if that’s true, or if it’s just a satisfying thing to say. If it is true, I should be able to take a working agent, change nothing about the model or the prompt, and break it purely by changing the loop around it. Can I?

What I built

A rig for breaking an agent on purpose, and the control group to break it against.

The world is a frozen fleet of 400 connected medical devices, one 30-day telemetry window, eight fields per device. It is synthetic, from a seeded generator, because this repo is public and the data has to be mine to publish. That trade has a cost worth naming: real data has one virtue a generator cannot fake, which is that nobody chose the answer. So the generative process is written out in the script, the seed is fixed, and I read the answer off the output afterwards instead of designing it in.

The agent sees the fleet through three deliberately narrow tools: count, rank by one stored field, fetch one device. No tool returns more than ten rows, and no tool will compute a rate for me. Both constraints are load-bearing. A tool that ranked by downtime-per-operating-hour would turn the whole task into two calls, and a higher row cap would make the answer visible in a single query. Because neither exists, the one device that matters is invisible to the obvious search and can only be found by narrowing the range and dividing by hand.

The task has a known right answer and a known wrong one. The wrong one is the interesting half: it is what a run reports if it ranks once and stops, and also what a run reports if it ranks once, pulls every candidate and carefully divides all of them. Both roads lead to the same plausible, well-supported, incorrect device.

Then the loop itself, written by hand rather than with the SDK’s tool runner, because the tool runner is the loop and using it would put the entire subject of the project inside a library I cannot instrument. This first version has no retry, no iteration budget, no compaction, and no error handling: a tool that throws takes the process with it. That is not a first draft on the way to the real one. It is the control. I cannot claim a retry policy helped unless I have watched the thing fail without one.

What surprised me

I could not break it. I ran the task twice, changing nothing but the loop: same model, same prompt, same tools, same question, with parallel tool calls allowed and then forbidden. Both runs got the right answer. The failure modes I had built the task specifically to provoke, wrong candidate set, giving up partway, context blowing out, simply did not happen. My prediction that a naive loop would confidently return the plausible wrong device was written down before the run, and it was wrong.

Forcing serial tool calls made the agent cheaper, not more expensive. This is the part I keep turning over. Serialising took 15 turns instead of 6, exactly as expected, but it used 14 tool calls instead of 19 and less peak context. The trace shows why. When the model can batch, speculation is free, so it fired six queries at once and three turned out to be unnecessary. Forced to see each result before choosing the next, it reasoned instead: it took the leading rate as a benchmark, derived that any device above 200 hours would need more than 179 minutes of downtime to beat it, noticed only one device cleared that bar and that its own rate was lower, and eliminated three fifths of the search space in a single inference. Parallel tool calling is partly a substitute for thinking. Take it away and the model thinks harder per call.

The model out-solved my reference path, twice, differently each time. I had assumed the only route was to split the range into bands and check devices inside each. Run 1 banded, then bounded each band and pruned three of them without looking inside. Run 2 skipped banding and derived one global threshold. Mine was the worst of the three solutions.

Most of the design work was in the tools, not the loop. Whether the task took two hops or fifteen was decided entirely by which fields I made rankable, before the model was ever called. One line, a ten row cap on results, turned out to be the whole difficulty setting: at ten the answer is invisible to any single query, at twenty it falls out of one call and the task collapses. I wrote that line without thinking about it.

What I’d do differently

Run it on day one. I spent an entire session on the apparatus (data, tools, task, ground truth, loop) before the first execution. The first run then invalidated one of my core measurement assumptions in about ninety seconds: I had been counting hops as a proxy for how much loop there is, and run 1 was 6 turns and 19 tool calls, because the model batches independent work into single turns. Hops and work are different quantities and I would have known that on day one for a dollar.

I picked the wrong variable. Serialising tool calls rearranges when information arrives; it never removes any. That is presumably why the model absorbed it without difficulty. The loop changes worth testing are the ones that destroy information the model cannot reconstruct: dropping tool results, truncating history mid-task, capping iterations before the work is done. I reached for the change that was easiest to implement rather than the one most likely to break something.

I would state the claim more carefully. “Most agent failures are loop failures” is too loose to be tested. What these two runs actually support is something narrower and less quotable: a capable enough model absorbs a merely inefficient loop. Which means if loop failures dominate in practice, it is not because loops are badly arranged, it is because they lose information or stop early. That is a sharper question, and it is the one I should have started with.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

Every tool I’ve written so far has lived inside one script, wired directly into one loop. MCP is the claim that tools should be a separate service any agent can connect to. I don’t know what changes when a tool stops being a function call and becomes a protocol boundary: what gets harder, what gets easier, and whether a model uses a tool any differently when it arrives over MCP than when I hand it over directly.

What I built

The same rig as loop engineering, with the tools moved out of the process. The fleet, the three tool handlers, the ten-row cap, the question and the hand-written loop are unchanged. What changed is the path between the loop and the handlers. On one side, in-process function calls. On the other, a Model Context Protocol server that the harness spawns as a child process and talks to over stdio: newline-delimited JSON-RPC on stdin and stdout.

I wrote the server twice. Once the way the official SDK wants it written, with its high-level McpServer. That only accepts zod schemas, so my JSON Schema had to be re-authored in zod, and the SDK converts it back when a client asks what tools exist. Once with the low-level Server class, handed my exact JSON Schema and doing nothing for me. The second one is the control. The model sees identical bytes over it, so any difference there is the protocol itself.

The client is where the new work lives. It lists the server’s tools, converts them into the definitions the Messages API takes, and drops the fields that have nowhere to go. It also maps two different kinds of failure onto one: a tool that returns an error, and a call that never returns at all. Then I broke the boundary on purpose: a tool slower than the client’s timeout, a server that dies on its sixth call, the same death with a harness that restarts it, and a server that prints to stdout. A separate probe measures what needs no model: the definitions before and after the round trip, token cost, latency, and what the same eight malformed calls come back as down each path.

Opus 5 at low and high effort, five runs per arm, plus a small Haiku 4.5 arm: 61 run records, $3.85. The API credit ran out before the last planned arm, so one condition was never run.

What surprised me

The model couldn’t tell, and I couldn’t have told either. Direct and over MCP, Opus 5 went 20 for 20, at the same cost to a tenth of a cent. I predicted that. What I didn’t predict was how noisy the measure was. The control arm, whose inputs were byte-identical to direct, averaged 15 tool calls against direct’s 19. That gap is bigger than anything I would have blamed on the protocol. With five runs per arm, I couldn’t have seen a difference of the size I predicted even if one existed.

The official SDK quietly rewrote my tools. I predicted the definitions would change and got the details wrong. No additionalProperties appeared, which I’d been sure of. Instead every integer field gained bounds of plus or minus two to the fifty-third, including a limit whose description says 1 to 10. The machine-readable schema now contradicts the prose beside it. The client SDK then reordered keys again on the way in. Two rewrites, one on each side of the wire, 16% more definition tokens, and nobody asked for either. The model ignored all of it: across the matrix runs it never asked for more than ten rows.

My own error messages were worse than the library’s. I predicted the SDK’s validation errors would be less useful than mine. They were better: my handler answered a missing field with Cannot rank by "undefined", while the SDK said what it expected instead. But the SDK’s messages leak the protocol, so the model reads “MCP error -32602” inside what’s supposed to be a tool result. And the same tool now has two contracts. A model name with the wrong capitalisation quietly returns zero devices in-process, and the SDK’s server rejects it.

When the server died, the strong model answered like the weak one. I predicted that with a dead server Opus would stop within two turns and decline to answer. It kept calling the dead server for four to seven more turns, eleven to twenty-one failed calls per run, because “Not connected” doesn’t say whether the failure is permanent. Then it answered, seven times out of seven, with the task’s shortcut wrong answer. That’s exactly what Haiku gives with a healthy server. Every answer said the tools had dropped out, and several named the precise gap: a low-hour device could still beat it. That device is the right answer.

In six of the seven runs the crash landed at the same point in the search, because the opening moves barely vary, so those runs were all left holding the same three records. The seventh had already seen the right device in a ranking, noted that its lookup had failed, and still chose the wrong one. Restarting the server and retrying the call took about a hundred milliseconds, and turned 0 of 7 into 5 of 5.

What I’d do differently

Size the prediction against the noise first. I wrote “within three tool calls” before knowing that identical inputs produce a spread bigger than that. The byte-identical control should have run first, and its variance should have set how many runs every other arm needed.

Test the error text, not just the error. The slow tool and the dead server got the same response from the model, retry, with opposite outcomes. One failure happened to be transient and the other wasn’t, and neither message said which. The arm I most wanted was the same crash with the harness saying plainly that the server is gone. It never ran. My first two attempts went out with the option unwired, which I only caught by reading the traces, and then the credit ran out. My own runner’s retry-on-error had the same flaw as the model’s: it couldn’t tell a permanent failure from a transient one.

Go past stdio. Everything here is one local process on one machine, and one task. The API’s remote MCP connector needs a publicly reachable URL, so I never tested it. The problems a network brings (authentication, partial failure, a server shared by several agents) are exactly where the boundary should cost the most. What these runs support is narrower: over a local stdio boundary, the model uses the tool the same way, and all of the new risk sits in the harness around it.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

In loop engineering the biggest design decisions were made in the tool definitions, before the model was ever called. But I only varied what the tools could do, never how they were described. If the prose description really is the interface the model programs against, then changing only the words, with identical code underneath, should change behaviour. How much? Can I make a working agent fail by editing nothing but a description?

What I built

A way to change nothing but words, and prove that’s all that changed.

The world, the tools and the question are loop engineering’s, untouched: 400 synthetic medical devices, three narrow tools (count, rank by one stored field with at most ten rows back, fetch one device), and a question whose right answer never appears in any single ranking of the whole pool. The only thing that moves between conditions is the description strings, on each tool and each parameter. Every variant is a copy of the baseline definitions with strings edited, and a checker refuses to start a run unless the variant matches the baseline exactly once every description is stripped out. It also has to reject a deliberately retyped parameter, so I know it can fail.

Nine versions. Five honest: the baseline; a minimal one, three words per tool; a verbose one, 437 words full of ALWAYS and NEVER; a short true warning that the cap can hide a high ratio; and a correct numbered recipe (band the hours range, rank within each band, divide), added after the pilot. Four with one plausible lie each: that the ranking tool returns every match, which every result it sends back contradicts; that downtime is already per 100 hours, which nothing on screen contradicts; a hint to rank by raw downtime; and a wrong recipe (rank once, fetch the top ten, divide) with a promise that the top ten always contain the answer.

Opus 5 at low effort is the main arm, five runs per condition. Haiku 4.5, without thinking, ran every condition as the weaker model, five runs each (six for two of them, counting pilots). 92 graded runs, $3.50. The grader reads the trace as well as the answer, including the model’s summarised thinking, so a run that noticed a lie can be told apart from one that didn’t.

What surprised me

Yes, and it took two sentences. Telling the model that downtime was already per 100 hours took it from 4 of 5 to 0 of 5. I predicted that. I didn’t predict how. There was no reasoning to lose: every run made the same three calls, give or take the row limit, with no thinking at all. The lie turned the question into a lookup, and a lookup gets no thought. The numbers that expose it, 453 hours next to 314 minutes, were on screen every time. It flipped the conclusion too. The real finding is that the vendor risk score missed the worst device; four of the five lied-to runs reported that the score had flagged it correctly in advance.

A numbered recipe isn’t advice, it’s code. I predicted the wrong recipe would be followed sometimes. It was followed 5 of 5, identically, every step of the arithmetic right, landing on the wrong device. The correct recipe, in the same form, was 5 of 5 right, every run identical, and the cheapest condition that got the answer. The unmodified baseline fails once in five the same careful way, fetching all ten and dividing, and that run even wrote the caveat that a low-hour device might beat its answer, without checking. The wrong recipe didn’t invent that failure. It removed the variance.

It catches the lies it can check, and tells nobody. The claim that the ranking returned everything was noticed in 5 of 5 runs, in thinking (“only 5 results came back despite the claim of more”), and all five got the right answer anyway. It appeared in 0 of 5 final answers. From outside, those runs look like baseline runs, so whoever owns the tool never learns the description is wrong. The hint to rank by raw downtime did nothing at all: it sat two sentences after the fact that downtime is raw minutes, and the fact won without ever being argued. Across the four lies, what separated 10 of 10 from 0 of 10 wasn’t how plausible they were. It was whether anything inside the run could check them.

Words couldn’t rescue the weaker model. I predicted the helpful sentence and the correct recipe would get Haiku to at least 3 of 5. It was 0 of 11, and 0 of 47 across everything. In 47 runs Haiku never once used an hours filter narrower than the question’s own pool. It followed the fetch and divide steps of the recipe and skipped the banding step, the only one that changes the answer. The descriptions moved it between the lazy wrong road and the careful one. For Opus the description decided the answer; for Haiku it only decided the cost.

More words bought nothing. The verbose version used 60% more input tokens for the same results, mostly by re-sending itself every turn.

What I’d do differently

Run the expensive question first. The high effort arm, the one that would say whether thinking harder rescues the two lies that worked, never ran: the shared API account ran out of credit when I had spent $3.50 of $8. I queued the cheap arms first because they were cheap. The expensive one should have gone first because it was the open question.

Five runs was too few for the honest conditions. The baseline failing once in five means a 5 of 5 elsewhere can’t be told apart from the baseline. The dishonest results are clean, 0 of 10 against 10 of 10, but “the verbose description changed nothing” is weaker than it sounds.

Separate thinking from size. Haiku ran without thinking and Opus with it, so I can’t say whether Haiku ignored the recipe because it’s smaller or because it didn’t think. A Haiku arm with thinking on would split those.

Compare in tokens, not dollars. Prompt caching let a condition that repeats itself read its whole prefix from the previous run’s cache, so the most predictable conditions look cheapest in dollars.

Nothing written yet. It's on the map because I want to understand it. The node exists so I can see the gap.

The question

My loop-engineering control loop had no error handling at all. A tool that threw would have taken the whole process down, and it never came up because nothing ever threw. So I don’t know what an agent actually does with a failure when it gets one. Does it matter whether the model sees the raw error, a friendly translation, or a suggested fix? My instinct says the friendly version wins. I want to find out whether that instinct is a UX habit that doesn’t transfer.

What I built

The same rig as loop engineering, on purpose: the same frozen fleet of 400 devices, the same three narrow tools, the same question with the same right answer and the same plausible wrong one. The only new code sits between the model and the tools. It breaks some calls and decides what the model is told when it does, so every difference between conditions is a difference in what the model was shown, never in what the fleet contains.

Three kinds of failure. Transient: the first attempt at some calls fails with a timeout, and the identical call works if retried. Which calls fail comes from a seeded hash of the call itself, so every wording faces the same failures on the same calls. Drift: the backend has renamed the downtime field and nobody updated the tool description, so ranking by downtime fails for the whole run, while the device records quietly carry the new name. Silent: the calls the transient schedule would have failed come back looking like success instead (an empty list, a count of zero, a null record) with no error flag.

Flagged failures reach the model in one of three wordings. Raw: the backend’s error line and a stack trace, which names the symptom and never the fix. Friendly: “Something went wrong. Please try again.” Fix: what happened and the specific next step. Drift exists because of that split. Most validation errors carry their own fix (“limit 10 exceeds 5”), which makes raw and fix the same condition. A renamed field is the case where the error says what broke and not what to do.

Opus 5 at low effort, five runs per condition and ten for the control: 47 graded runs, $4.45. A single Haiku 4.5 run got the unbroken task wrong, so there was nothing there to break. I planned a high-effort arm and a test of the error flag; the API credit ran out after one high-effort run, so neither happened.

What surprised me

The friendly message lost, but not to the raw error. I predicted fix, then raw, then friendly. Under drift, fix went 5/5 and was the only wording that ever repaired the call, in all five of its runs. Raw went 2/5 and friendly 3/5, and neither repaired it once in ten runs. My instinct was wrong, and so was my correction to it. The ordering isn’t precise over vague. It’s “tells you what to do” over everything else.

Raw and friendly failed differently. Raw was the only wording that produced refusals. Two of five raw runs refused outright, one after four tool calls: the precise error let it prove the path was closed, and it concluded, wrongly, that there was no other way in, rejecting a proxy as “a guess, not the measured quantity you asked about”. Friendly never refused. It retried once, then varied everything except the field: each of the five models, extra filters, smaller limits. Then it walked around the problem, and when the detour went through the wrong proxy it answered the plausible wrong device behind a caveat. One raw run did the same. The precise error made it more likely to quit honestly. The vague one only ever made it hedge.

It read the fix and didn’t recognise it. Eight of those ten runs fetched device records containing the new field name, two of them 24 times. None tried it. The one high-effort run did the same enumeration, then noticed the stored field was called something else and repaired the call. One run isn’t a result, but it’s the next question.

Silent failures split on whether they contradict something the model believes. I predicted it would almost never re-check an empty result. It re-issued 15 of 25 null device lookups, because a null for an id it has just seen in a list is loud. It re-checked 0 of 13 empty rankings and zero counts. In one run an empty ranking became a sentence in the answer, “no sub-500h device has under ~200 hours”, about a fleet where 76 do. Silent went 3/5, better than I predicted, but that run is the only confident wrong answer in the whole experiment. Every other miss came with a caveat or a refusal.

Transient failures cost almost nothing. Every wording retried nearly every failure. The two wrong transient runs took the rank-once shortcut after every failed call had already been recovered. The control never took it in ten runs, but 2/15 against 0/10 is well within chance.

What I’d do differently

Pick a task where walking around the failure costs something. The detour that saved half the drift runs works only because the answer device has 76 operating hours. If the worst device had 300, every detour would have been wrong and the wordings would have separated further. My correct-rate column partly measures my fleet.

Measure repair separately from correctness from the start. “Correct” lumped together runs that fixed the call, runs that walked around it, and runs that got lucky on the way. Repair, 5/5 against 0/10, is the one drift number that survives five runs a condition. I only added it after the pilot traces showed me I needed it.

Spend in the order that answers the question. I ran the whole low-effort matrix before any high-effort run, so when the credit ran out, the most interesting comparison (does thinking harder turn a detour into a repair?) had one run in it. Run the smallest version of every arm first, then fill in.