The value the model never held.

A user asks our agent for a status report: filter a project’s work items to the open ones and produce a PDF summary. The items export is 2.1 MB of JSON. In a naive harness that value enters the model’s context twice: once when the query tool returns it (~500k tokens of input — if it fits at all), and again when the model passes the rows to the PDF tool, because the only way to hand a value to the next tool is to write it out token by token. That second copy is the expensive kind — output tokens — and the model is now a lossy, non-deterministic wire: it will truncate, “fix,” or hallucinate rows in transit.

Part 1 kept the domain model out of standing context; part 2 kept the tool catalog out. Both were fixed-size wins. This one is unbounded: the data never enters context at all. The model works with references, and the harness resolves them server-side at the moment a tool actually needs the bytes.

The vocabulary

A reference is not a URI or an opaque handle. It’s a small typed JSON object the model can read, and — in two of four cases — write:

{ "$ref": { "type": "Message", "seq": 42, "part": 2, "field": "result.value" } } the envelope — replaces any top-level property, or the whole input the address space: Message · Blob · (derived, server-only) which message in the transcript — omit for the current turn which part of it dot-path within Message seq · part · field Blob key · or source JsonPointer source · path JsonQuery source · path · filter the model may author these server-only recipes — the model forwards, never writes
The reference the model sees, and the four-member union behind it. Message and Blob references address things the model has legitimately seen named; JsonPointer and JsonQuery are derivation recipes that only the server may construct.

Any top-level property of any tool call can be replaced by { "$ref": … }, or the entire input can be one. A Message reference addresses a part of the transcript itself — message 42, part 2, field result.value — which means the transcript doubles as an addressable store: anything a tool ever returned is a coordinate, not a copy. A Blob reference addresses stored bytes by key or through another reference. The two derived forms — JsonPointer (one node of a JSON document) and JsonQuery (an array filtered by field equality or substring) — are recipes: the model can forward one it received, but the input parser rejects any it tries to author. That asymmetry closes an injection surface — the model cannot fabricate a pointer into data it was never shown — and it keeps recipes an implementation detail the harness is free to change.

How values leave context

The interesting design choice: there is no size threshold. Whether a result passes by reference is decided by type, at two chokepoints.

First, any Blob in a tool result is intercepted on the way out. The bytes are streamed to an object store under agent/<agentId>/<key> — chunk by chunk, never materialized — and the result field is rewritten in place to a typed marker. Second, when the provider serializes the result for the model, the marker becomes:

{
  "opaque": true,
  "value": { "$ref": { "type": "Message", "part": 0 } },
  "size": 2214731,
  "format": { "mediaType": "application/json" }
}

Metadata, and an address. Never bytes, never even the storage key. A 2 MB file and a 2 KB file cost the same ~120 characters of context.

Structured results — plain objects — inline as-is, which sounds like the loophole that ruins everything, until you look at how the file-reading tools are shaped. None of them can return an unbounded value. read_json and query_json return a preview + reference pair: the preview is paginated and capped — long strings elided, child and item counts bounded — and alongside it comes a derived reference to the complete, exact value — {type: "JsonPointer", path: "/items"} or {type: "JsonQuery", path, filter}. Even that recipe is hidden: the model sees a Message reference pointing back at the result’s own value field. The test suite asserts the string "JsonPointer" never appears in what the model reads.

So the model always holds enough to decide — shapes, counts, first rows — and an address for the whole thing, evaluated only if some tool eventually resolves it.

The flow, end to end

MODEL CONTEXT — EVERY CHARACTER HERE COSTS TOKENS, EVERY TURN query_json({ source, filter }) the call — a few dozen chars preview: 10 of 148 items value: { $ref } ≤6,000 chars + a ~60-char ref generate_pdf({ rows: { $ref } }) forwards the ref, not the rows ~60 chars the context boundary — whatever crosses it, costs read + filter, server-side bounded preview + derived $ref resolver re-runs the recipe items.json — 2.1 MB in the object store agent/<agentId>/<key> filtered where status = "open" → 148 rows; the recipe {path, filter} is kept, not the rows ≈500k tokens if inlined — never crosses resolve → execute full 148-row array handed to generate_pdf; result pdf stored as a new blob → { opaque, size, $ref } bytes stay below HARNESS + OBJECT STORE — BYTES ARE FREE HERE Round trip: 2.1 MB brokered by the model for the cost of one preview and two ~60-char references.
The status-report scenario. The 2.1 MB export is read, filtered and re-read entirely below the context boundary; the model orchestrates it with one bounded preview and two ~60-character references. In the harness's own test suite this round trip is asserted with the document provably absent from the model's view.

The step that still surprises me is the third arrow: when generate_pdf receives rows: {$ref}, the resolver walks Message → result field → recovers the stored JsonQuery recipe → re-reads the blob → re-applies the filter — and hands the tool the full 148-row array. The recipe is lazy: it was never evaluated when query_json returned, only now, at the moment of use. The model brokered a value it never held. Resolution is recursive with two independent caps — one at validation, a tighter one at resolution — so Blob(source: JsonQuery(source: Message)) chains are legal but bounded.

Cheaper is also more correct

The token arithmetic gets the headlines, but the correctness argument is what makes references non-negotiable. When values travel tool-to-tool through the model, the model is part of the data path: every row is re-generated token by token by a sampler. It abbreviates long fields, “fixes” values that look inconsistent, rounds numbers, drops a row and keeps going — none of which is a bug in the model. That is simply what generative transport is.

By reference, the model is only the control path. It routes addresses; the data path is deterministic server code. The 148 rows that reach generate_pdf are byte-for-byte the filter’s output, not the model’s recollection of it. And the failure modes improve along with the fidelity:

  • A stale pointer throws — “Derived JSON path … no longer exists.” — loudly and immediately, instead of silently shipping plausible-looking rows.
  • The model cannot author a derivation recipe, so it cannot point a tool at data it was never shown; a fabricated reference fails validation instead of fabricating data.
  • The preview and the reference travel together, describing the same exact value — there is no gap between what the model inspected and what the next tool receives.

Token savings you can buy with a bigger context window. This property you can’t: the only way the model corrupts a value in transit is if the value transits the model.

The plumbing that makes it declarable

For the model to pass a reference anywhere, every tool’s schema must say so. The harness rewrites each tool’s input schema so that every top-level property accepts anyOf: [original, $ref]. Two details echo part 2’s discipline:

  • The alternative is attached per property, never at the schema root — the Anthropic API rejects oneOf/anyOf at the top level of input_schema, so the envelope design bends around a provider constraint.
  • The rewritten schema is cached in a WeakMap keyed by the original parameters object — so the declaration bytes are stable across turns, and the prefix cache keeps hitting.

Tools that want laziness inside their implementation opt out of eager resolution: read_json’s ref parameter receives the reference object itself rather than resolved bytes, so the tool can stream from the store on its own terms. Everything else gets its $ref arguments resolved before execute runs — tool authors mostly never know references exist.

One envelope, one union, one resolver

This machinery is the product of a deliberate unification. Before it, the codebase had accumulated three ways to say “that value, over there”: bare {seq, part} objects for message parts, {key, ...metadata} markers for staged blobs, and positional seq="9" part="2" attributes in attachment tags. Each had its own parser and its own resolution path. The unification collapsed them into the single $ref envelope over the single typed union, resolved by one recursive resolver — and the boundary now rejects the legacy shapes, with tests pinning the rejection. The old forms survive only deep inside resolution for backward compatibility, guarded by a pointed comment: generic value resolution must not reinterpret an ordinary domain object merely because it happens to contain a key field.

That’s the unglamorous part of harness engineering: the token win was available with any of the three ad-hoc shapes. The unification is what makes the mechanism explainable to the model in one paragraph of protocol prompt — and models are much better at one rule than at three.

That closes the loop this part opened: the model routes data by address, the harness moves bytes the model never sees, and the token bill and the error rate drop together. But the standing context still has rows we haven’t examined — and they’re different in kind. The ontology, the catalog and the data are all knowledge we shipped. The agent also picks things up along the way: the user’s preferences, a project’s quirks, corrections it shouldn’t need twice. That accumulating knowledge needs the index-and-payload treatment more than anything else — it grows without bound by design — and it’s the subject of part 4. What users deliberately teach the agent is a different mechanism again, and it closes the series in part 5.