← research lab

Mac-local specialist field guide

How to build an agent trajectory dataset

Build an agent trajectory dataset with typed events, provenance, privacy filters, leakage-safe splits, and outcome-based evaluation.

An agent trajectory dataset records more than the final answer. It preserves the task, observations, tool calls, tool results, decisions, corrections, and outcome so a specialist can learn the behavior that produced success. The dataset is only useful when its provenance, privacy, and evaluation boundary are explicit.

Define the training unit

Choose whether one example is a full episode, one decision point, or a correction pair. Preserve the original task and enough state to understand each action. Avoid training on hidden evaluator feedback that the runtime will never receive.

Capture typed events

Normalize observations, assistant reasoning that is permitted for training, tool names, arguments, results, errors, and final outputs into a versioned schema. Keep raw source references so the transformation can be audited. Do not flatten every event into an undifferentiated chat transcript.

Filter for useful behavior

Remove duplicates, leaked tests, private values, malformed tool calls, and trajectories without an authoritative outcome. Separate successful examples from correction data and hard negatives. Preserve the reason an example was accepted or rejected.

Split by source and task family

Random row splits can leak nearly identical episodes across training and test sets. Group by repository, task template, session, or source artifact before splitting. Freeze the holdout before tuning the training recipe.

Make one episode inspectable before converting it to training text

This synthetic source record keeps an observation, tool call, result and final answer distinct. source_group identifies related sessions or a repository snapshot; task_family groups templated variants. Keep the authoritative outcome, provenance and review decision beside the events. This is an illustrative manifest, not an MLX training-row format.

{"id":"episode-1","split":"train","source_group":"fixture-repo-a","task_family":"list-files","outcome":"checked-success","source":"synthetic-fixture","events":[{"type":"observation","payload":{"request":"List files in /demo"}},{"type":"tool_call","payload":{"name":"list_directory","arguments":{"path":"/demo"}}},{"type":"tool_result","payload":{"entries":["notes.txt"]}},{"type":"final","payload":{"text":"notes.txt"}}]}

Before export, review paths, file contents, credentials and personal data; retain only material you have rights to use. Keep failed episodes as labelled negatives or corrections rather than silently teaching them as successes. Do not add private chain-of-thought or hidden evaluator feedback. Tool payloads and stop behavior need their own contract checks.

Check grouped splits before selecting a recipe

Download check-trajectory-splits.py. Put one manifest episode on each line of episodes.jsonl, with nonempty train, valid and test splits. Run:

python3 check-trajectory-splits.py episodes.jsonl

The check requires typed nonempty events, unique IDs, exact-event deduplication and disjoint source groups and task families across splits. This intentionally strict policy prevents a templated task family from appearing on both sides of the evaluation boundary. If your intended test measures new examples within a known family, define and document that different grouping policy before tuning; do not weaken it after seeing scores.

The checker was exercised with valid synthetic splits plus source overlap, family overlap, duplicate episodes, duplicate IDs, empty splits and malformed events. It does not detect paraphrases, prove privacy review, verify tool execution or convert the manifest to a trainer schema. Inspect the source-pinned MLX tools data format before conversion, preserving tool-call identifiers and the model’s chat-template contract.

Train the narrowest useful target

A trajectory dataset can support supervised fine-tuning, preference pairs, distillation, routing, or an evaluator. Choose the objective from the observed failure. If the model over-edits, an objective that increases edit pressure is the wrong response.

Evaluate the behavior, not imitation

Measure whether the specialist selects the correct tool, produces valid arguments, reacts to tool results, stops appropriately, and completes the task. Also measure unsupported actions and breadth regressions.

PostTrainLLM includes trajectory conversion, correction-to-data, synthesis, quality filtering, deduplication, tool-calling evals, and factory-run evidence. Data and model training remain local and owner-controlled; public report cards should expose only safe, reviewed evidence.

CTA: Continue to the specialist-model workflow or inspect the factory documentation.