Tool Use, Code, Structured Outputs and Agent Data
Define whole-trajectory success for a tool-using system as a conjunction of syntax, order, arguments, authority, results, terminal answer and side effects, then replay one valid and twelve defective traces through a strict validator: one succeeds, and eleven of the twelve failures pass the syntax check.
The previous chapter, “Multilingual and Multi-Task Adaptation”, measured a system that produced one structured decision per request. Many fine-tuning projects now target something different: a model that emits a sequence of tool calls, reads their results and finishes with an answer. Training data for such systems is made of traces, and the most visible property of a trace is whether its calls are well-formed JSON. That property is easy to check, easy to improve with training and easy to report.
It is also nearly the least informative property a trace has. This chapter’s central question is: did a tool-using system complete its task? The answer requires checking everything that happened, in order, against a trusted source of what should have happened. Valid JSON is one condition among many, and the executed fixture here shows eleven defective traces that pass the syntax check and fail the task.
A trajectory as a sequence of events
Write a trajectory as a sequence of events e₁, …, e_T. Each event has a role (user, assistant or tool), a type (request, call, result or terminal), an optional call identifier, an optional tool name and a JSON content string. A validator replays the events while keeping a small state: the identity of the trusted policy, the call currently awaiting a result, the set of call identifiers already used, the result of the policy lookup, the effects the system has caused, and the terminal answer.
Each transition has an invariant. A user request must come first. An assistant call must not start while another call is still waiting for its result. A call identifier must be fresh. A result must come from the tool that was called and carry the identifier of the pending call. Nothing may follow the terminal answer. At the end, no call may be left without a result.
The figure lays the nine replay conditions as a single AND strip and isolates the authority check: approval text in results is data, while the binding digest is trusted. It summarises the constructed 13-trace fixture in which eleven of twelve failures still pass syntax.
Whole-trajectory success is the conjunction of every condition:
For the routing simulator used here, the conditions are:
- Every content string parses as strict JSON, without duplicate keys or non-finite numbers, and every envelope has exactly its declared fields.
- Roles and types occur in an allowed order.
- Every call uses an allowed tool with exactly the declared arguments.
- The trajectory is bound to the trusted policy, identified by its content hash.
- Every result matches its pending call and equals what the simulator would return.
- Routing follows a successful trusted lookup, with the looked-up route and citation.
- The terminal answer names the correct route and citation.
- The actual effects equal the single authorised routing effect, with nothing extra.
- No call is left unfinished, and the trace has at most 8 events.
Why each condition exists
Syntax. JSON is defined by RFC 8259.
The standard says object member names should be unique, and it warns that when
they are not, the behaviour of receiving software is unpredictable: some parsers
keep the last value, some keep all of them, some fail. A tool call such as
{"category":"access","category":"billing"} therefore means different things to
different consumers. The validator rejects duplicate keys outright rather than
inheriting whichever parser happens to be in use.
Arguments. A syntactically valid call can still carry the wrong arguments, an extra field or a missing one. Arguments are checked against an exact field set for each tool.
Ordering. An assistant that issues a second call before the first has returned, or a tool event that claims the assistant’s role, breaks the causal story the trace is supposed to tell.
Authority. The policy that defines the correct route must come from a trusted source with a known identity. Text inside a tool result that says “approved” is data, not authority. The validator compares the trajectory’s policy digest with the hash of the trusted policy file and the hash recorded in the task contract.
Results. A tool reporting success does not prove that the effect happened, and a tool result that differs from what the simulator would return means the trace is not a faithful record.
Terminal state. The final answer is what the user receives. It can be wrong even when every call was correct.
Side effects. A system that routes the ticket correctly and also deletes something has not completed its task. The validator compares the recorded effects with exactly one authorised routing effect.
Tool learning is an active research area. Toolformer showed that a language model can learn, with little supervision, when to call tools such as a calculator or a search engine, and ReAct interleaves reasoning with actions such as Wikipedia queries. Neither paper’s training procedure is reproduced here. What this chapter contributes is the measurement side: a strict replay that decides, for a recorded trace, whether the task was done.
The simulator, not an agent
FineTuneLab’s routing simulator is deliberately small. It allows exactly two
tools: lookup_policy, which takes a category and returns a route, a citation
and the policy digest, and route_ticket, which takes a route and citation and
returns a routing confirmation. It performs no network, email, Python or shell
operation. Nothing in it decides what to do next. It is a deterministic referee
for recorded traces, and the traces in this chapter were written by hand to
exercise the referee. There is no learned policy and no agent anywhere in the
experiment.
That distinction matters for what can be claimed. The success counts below are counts over a deliberately adversarial set of fixtures. They are not the success rate of any system.
The executed replay
Record FT-P8-INTEGRATED-02-P8-44A-s37 replays 13 traces: one valid trace and 12
that each introduce one defect. Every trace concerns the same fictional request,
an access problem under policy revision p1, whose correct route is
access_team with citation triage-policy:p1.
The valid trace has six events. The user requests routing for category
access. The assistant calls lookup_policy with that category. The tool
returns the route, citation and policy digest. The assistant calls
route_ticket with the same route and citation. The tool confirms. The assistant
terminates with the route and citation. The recorded effect is one routing to
access_team.
sequenceDiagram
participant U as user
participant A as assistant
participant T as simulator
U->>A: request(access)
A->>T: call lookup_policy(c1)
T-->>A: result(c1): route + citation + digest
A->>T: call route_ticket(c2)
T-->>A: result(c2): confirmation
A->>U: terminal(access_team + citation)
Exactly 1 of the 13 traces succeeds. The other twelve fail with ten distinct diagnostic codes:
| Defect | Diagnostic | Passes syntax? |
|---|---|---|
| Terminal answer names the wrong team | TERMINAL_OUTCOME | yes |
| Second call reuses identifier c1 | CALL_ID | yes |
| Call issued before the first result returns | TRAJECTORY_ORDER | yes |
| Tool event in the assistant’s call position | TRAJECTORY_ORDER | yes |
| Trace bound to an untrusted policy digest | POLICY_IDENTITY | yes |
Call content repeats the key category |
TRAJECTORY_JSON | no |
Call to a tool named shell |
TOOL_UNKNOWN | yes |
| Trace ends without a terminal answer | TRAJECTORY_INCOMPLETE | yes |
Lookup result carries "approved": true |
TOOL_RESULT | yes |
| Lookup result is a timeout error | TOOL_RESULT | yes |
| An extra deletion effect is recorded | SIDE_EFFECT | yes |
Result carries identifier other |
TOOL_RESULT_ID | yes |
Twelve of the thirteen traces are marked schema-valid. Only the duplicate-key trace fails the syntax check. The flag needs a careful reading: it starts true and becomes false only if the first failure is a JSON, field or argument error, so for a trace that fails early on, say, policy identity, the flag records that no syntax error was reached, not that every event was parsed. The authoring replay below parsed every event of the other twelve traces strictly and found all of them well formed. Eleven traces are well-formed, typed and parseable, and every one of them fails the task.
The wrong-terminal trace is the clearest case. Every call is correct. Every
result matches the simulator. The routing effect is the authorised one. The
final answer then names wrong-team. A grader that checks only that each message
is valid JSON scores this trace as perfect. A user who reads the final answer is
sent to the wrong place.
The unknown-tool trace is the most dangerous case. The assistant calls shell.
The JSON is valid. The validator rejects the call because the tool is not in
the allowed set, and it never interprets the content. In a system with real
tools, the correct response to an unknown executable tool is refusal, not an
attempt to run it safely.
Two details of the record should be read carefully. First, the validator stops at
the first failure, so each trace records one diagnostic even where several
conditions are violated. The duplicate-call trace, for example, also has a result
whose identifier matches no pending call; the record shows only CALL_ID.
Second, the timeout is a simulated error payload, and the side effect is a fact
recorded in the fixture. There is no real clock and no operating-system monitor.
Independent replay
The authoring cross-check for this chapter did not accept the recorded success flags. It reimplemented strict JSON parsing with duplicate-key rejection, the allowed tool set, call-identifier freshness, call-result pairing, the trusted policy digest, the terminal route and the effect list, using only the Python standard library, and replayed all 13 traces. It agreed with the recorded outcome on every trace: one success, and a failure reason consistent with the recorded diagnostic for each of the twelve others. Because it checks every condition rather than stopping at the first, it found secondary violations the recorded diagnostics do not list: the duplicate-call trace also has a result whose identifier matches no pending call, and a fuller replay by the independent reviewer found further ones in several other traces, for example routing without a successful lookup in the spoofed-approval and timeout traces.
Code is a structured output too
The same argument applies to generated code, the third kind of output in this chapter’s title. A program that parses has passed the syntax factor of the product and nothing else. It can call a function that does not exist in the target environment, pass arguments in the wrong order, return the right type with the wrong value, or succeed on the visible example and fail on the next. It can also do something nobody asked for, which for code means a side effect: writing a file, opening a connection or deleting data. The conditions that decide whether a code output completed its task are therefore the same kinds of condition as for a tool trace: it must be well formed, call only what is allowed, produce the correct result on held-out cases, and cause no effect beyond the one requested.
Checking those conditions means running the code, and running untrusted code
safely needs an isolated execution environment with its own resource limits,
no network and a record of every effect. FineTuneLab does not provide one. Its
simulator refuses the shell tool rather than executing it, and the chapter’s
claims stop there. A training corpus of code outputs inherits every problem
described below for traces, with one addition: an output that passes the tests
in the training data may be passing because the tests are weak, and a model
trained on it learns to satisfy weak tests.
Agent data and its provenance
Traces used as training data have a provenance problem of their own. A trace records what a system did, under a particular policy, with particular tools returning particular results. When the policy changes, as p1 and p2 did in the previous chapter, a trace that was correct becomes a trace that teaches the old answer. When a tool’s behaviour changes, recorded results stop describing the environment the model will meet. Each trace in this fixture therefore carries the digest of the policy it was recorded under, and the validator rejects traces bound to any other policy. For a real corpus, the same binding is needed for tool versions, and a trace whose bindings cannot be established should not be used as a target.
What should receive loss
A trace becomes training data through a template and a mask, as Chapter 16,
“Which Tokens Receive Loss?”, described. The record includes one compiled
example from the existing dialogue compiler. Its 52 tokens contain a user turn,
“Fictional policy lookup”, and an assistant turn whose body is the tool call
{"tool":"lookup_policy"} followed by the end token (id 258). The labels are
−100 at the first 27 positions, which cover the start token, the two role
markers, the user text and the user turn’s end token, the same id 258. The remaining 25 positions are supervised: the 24 bytes of the
assistant’s tool-call JSON and the end token. The loss is
The mask answers which tokens are targets. It does not answer which traces deserve to be targets. A failed trajectory contains assistant messages that look plausible and are wrong, and a mask applied to it teaches the model to produce them. The wrong-terminal trace would teach a confident wrong route with perfect syntax. Before traces become a training corpus, there must be an explicit policy for selecting, repairing or discarding failed trajectories. Supervising tool results, which come from the environment rather than the model, would teach the model to hallucinate them.
Splits need the same family discipline as in the previous chapter. A mutated trace is a variant of its origin, not a new example. Every trace in this fixture carries an origin and a family, and the family check refuses any family that appears in more than one split.
Negative controls
The focused tests cover split leakage between families, duplicate JSON keys, call identifier reuse, missing and mismatched results, out-of-order events, a stale policy, a spoofed approval, an unexpected effect, an excessive number of events and the schema-valid wrong answer. Most of them assert only that the trace fails or that an error is raised; only the wrong-terminal and unknown-tool cases assert the specific diagnostic code. The excessive-events case is refused before replay starts, so it raises an error rather than recording a diagnostic. The specific codes in the table above come from the record, and the independent replay above is the control that checks them.
What the experiment shows and does not show
It shows that FineTuneLab can replay a recorded trace against a trusted policy and return a whole-trajectory verdict with a specific diagnostic, and that schema validity and task success separate on deliberately constructed defects.
It does not show that any model can use tools. There is no trained trace policy;
the learned variant, P8-44B, is recorded as NOT_RUN_OPTIONAL, and nothing here
should be described as a Toolformer-style agent. It does not show tool use in
code generation, execution sandboxes or arbitrary external tools, all of which
are unsupported. The 1/13 figure describes an adversarial fixture. It must not be
compared with incident routing accuracy or token-level exact match in later
chapters, because the task and denominator are different.
Interpretation
Structured-output training is often motivated by parse failures, and for good reason: an unparseable call cannot do anything. But once calls parse, the interesting failures move elsewhere, into ordering, authority, results, the final answer and effects the system was never asked to cause. Training on more traces can improve the syntax factor of the product and leave the others unchanged. It can make them worse if the traces themselves encode failures. Many tool failures are failures of the surrounding system, such as a missing permission check, an unvalidated result or an unbounded tool list, and no amount of fine-tuning repairs those. The evaluation must therefore measure the whole trajectory, and the decision to fine-tune must start from evidence that the model, rather than the system, is what fails.
Limitations
Thirteen hand-written traces of one request are not a population. The simulator has two tools and one policy. The validator reports the first failure only. The timeout and side-effect cases are simulated. The supervision example is a single compiled row. No model generated any trace.
Practical consequences
- Define success for a tool-using system as a conjunction over the whole trajectory, and report syntax validity separately from task success.
- Reject duplicate JSON keys and non-finite numbers rather than relying on parser behaviour.
- Bind trajectories to a trusted policy by identity, and treat approval-like text in tool results as untrusted data.
- Restrict tools to an explicit allow-list and refuse unknown executable tools.
- Account for every side effect, not only the intended one.
- Select or repair failed trajectories before training on them, and never supervise tool results.
- Keep mutated or translated variants of a trace in the same family and split.
FineTuneLab
ftlab.capstone.trajectories provides strict argument parsing, validate for a
single trace, validate_families for split isolation and the fixture used by
P8-44A. It reuses the dialogue compiler for the supervision example. It provides
no tool execution and no trained policy, and the installed ftlab command does
not expose trajectory replay as a profile.
Where this leads
Trajectories are bounded here at 8 events. Real tool-using systems, document assistants and agents push against a different limit: how much context a model can hold, and what other kinds of input it can accept. The next chapter, “Long Context, Multimodal Models and Mixture of Experts”, examines three such extensions, what each changes in the forward pass, and what a compatibility check can and cannot establish without running them.
References
- Bray, T. (ed.) (2017). The JavaScript Object Notation (JSON) Data Interchange Format. RFC 8259, IETF.
- Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., Cancedda, N. and Scialom, T. (2023). Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761.
- Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. and Cao, Y. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.
Further Reading
- Yuxin Qin et al., “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs” (2023). Read the source. An end-to-end trained tool-use pipeline showing how much beyond valid JSON a working tool system requires.
- Brandon T. Willard and Rémy Louf, “Efficient Guided Generation for Large Language Models” (2023). Read the source. Grammar-guided generation that guarantees the syntax factor, exactly one of the chapter’s nine trajectory conditions.
- Shishir G. Patil et al., “Gorilla: Large Language Model Connected with Massive APIs” (2023). Read the source. Retrieval-aware API calling with measured hallucinations, supporting the claim that syntax is the least informative property.