What agent trace divergence actually tells you

You run your agent 30 times on the same task. 18 pass, 12 fail. The traces look different. They branch at different points, pick different tools, choose different files. The hypothesis everyone reaches for is:

Those branch points are where the decisions matter. Find them, score them, turn them into training data.

That’s the dream. Free DPO pairs, generated from variance you already have, with no labeling cost.

I spent three weeks testing that hypothesis on SWE-rebench v2: 1,096 tasks, 12,854 runs, multiple models, multiple harnesses. I built a tool called moirai to run the experiments end to end.

Most of it doesn’t work the way you’d hope. But the parts that do are narrower and sharper than I expected, and the negative results are more useful than the positive ones.

The setup §

Each task has multiple runs with mixed outcomes. Each run is a sequence of tool calls: read(source), search(grep_targeted), edit(source), test(pass), bash(python), and so on.

To find divergence points, align the runs with Needleman-Wunsch. At each column, group runs by the step they took. Test whether the branch predicts outcome with Fisher’s exact, correct for multiple comparisons with Benjamini-Hochberg.

That gives you a list of statistically significant divergence points per task. Within a task, the signal looks real. The question is whether it generalizes, whether it’s uniform across tasks, and whether you can do anything useful with it.

Six questions, six experiments.

1. Does divergence correlate with outcome? (Weakly) §

Split each task’s runs into train/test halves. Find divergence points on train. Use them to score held-out test runs. Measure AUROC.

0.507 is a coin flip. Pooled is slightly better because large tasks dominate, but per task the divergence signal is essentially noise on held-out data.

Features do better, at a modest 0.056 above chance. Real but weak. Trajectory shape matters more than the exact branch choice.

Takeaway: The divergence signal that looks strong in sample collapses on held-out data. If you’re training on one task’s divergence and applying to another, you’ve got nothing.

2. Does reranking with it work? (Slightly) §

Second experiment: best-of-K selection. For each task, sample K=3 runs and pick the one with the best score. Compare to random selection and oracle (always pick a passing run if one exists).

Features capture 13% of the oracle gap. Divergence captures 10%. Both are real but neither is close to what a reward model or perplexity-based reranker would give you.

The number that matters is the oracle gap itself. 33 points of potential lift are sitting in the data. Current methods capture a small slice. The rest is locked up in task-specific structure that doesn’t generalize across families.

3. Where does it generalize? (The 25% with structure) §

I computed “captured oracle gap” per task family and got a range from -73% to +100%. Some families gain a lot from divergence-based scoring. Others lose. The aggregate mean of +10% hides it.

So I built a per-task score called structure from three equally weighted components:

• branch_gap: mean pass-rate spread at the top divergence points
• earlyness: how early in the trajectory those splits occur (early = more decisive)
• stability: how reproducible the divergence map is under run resampling

Kendall’s tau between structure and divergence reranking lift: +0.402 (p < 0.0001).

Split tasks at structure ≥ 0.20:

• High-structure (269 tasks, 25%): divergence scoring gives +12.5% reranking lift
• Low-structure (811 tasks, 75%): divergence scoring gives +0.4% lift

That’s the whole story. In one quarter of the tasks, the signal is strong. In three quarters, it’s drift indistinguishable from noise. Aggregating across both hides the fact that you have two regimes, not one.

Takeaway: Divergence-based analysis works on a minority of tasks with a detectable property: early, reproducible, outcome-correlated branching. For everything else, the runs look like samples from a diffuse distribution over successful paths.

4. Can you route between scoring methods? (Yes, modestly) §

If you know structure per task, you can pick the right scoring strategy per task instead of globally:

Use divergence on high-structure tasks, features on low-structure tasks. Beats both global strategies by +2pp.

It’s a routing rule over two heuristics. If you’re already running features or a reranker, adding structure-based routing is a drop-in that nudges the number up. If you’re not, this isn’t enough to justify building the infrastructure.

Takeaway: Structure score is a method router. You use it to decide which existing scoring strategy to apply per task. It is not a standalone scorer.

5. Can you use structure to allocate eval budget? (No) §

If some tasks have structure and some don’t, intuition says: spend more eval runs on the structured ones. They should need more samples to characterize. Use the rest of your budget uniformly on low-structure tasks.

That’s wrong, and it’s wrong for a reason that took me a day to see.

Kendall’s tau between structure score and task volatility (p(1-p)): -0.273 (p < 0.0001).

High-structure tasks anti-correlate with volatility. Why? Because structured tasks have decisive branch points: early decisions that lock in the outcome. That means pass rates cluster near 0% or 100%, which means low Bernoulli variance, which means fewer runs needed to characterize the rate, not more.

Simulate it. Three policies under fixed compute budget:

Uniform wins at every budget ≥ 2.0. The structure-aware policy plateaus because it keeps spending compute on high-structure tasks that already have stable estimates.

Takeaway: Structure and volatility are orthogonal axes. Structure predicts which scoring method works. Volatility predicts how much sampling you need. They are not the same latent variable, even when your intuition insists they should be.

6. Can you extract DPO pairs from divergence? (No, and this one isn’t new) §

The original dream: at each divergence point, the better-outcome branch is the preferred response and the worse-outcome branch is the rejected response. Feed those pairs into DPO. Train a better agent for free.

There are two problems.

Problem 1: the signal is too weak to justify the infrastructure. AUROC 0.556 with features and 0.507 with raw divergence means any preference pair you extract has heavy label noise. DPO is already fragile to label noise; pairs from this signal would train the model on mostly random distinctions.

Problem 2: someone already tried this. SWE-Dev explicitly tested preference learning (KTO, OREO) on SWE-bench trajectories and rejected it in favor of SFT on successful trajectories. All five public SWE-bench fine-tunes I looked at (SWE-Gym, SWE-Fixer, Lingma SWE-GPT, SWE-Smith, SWE-Dev) use SFT. DeepSWE is the exception and uses online GRPO, which is a different animal.

The structural reason it doesn’t work: preference pairs extracted from trajectory divergence are conditioned on observed actions, not on the latent state of the repository/task. Two runs that took different actions at step 7 might have been in subtly different states (files opened, context accumulated, model hidden state) that the extracted pair ignores. You end up training on correlations that are confounded by invisible prefix state.

Takeaway: Don’t extract DPO pairs from agent trace divergence on SWE-bench. SFT on successful trajectories remains the right move.

What’s left that’s useful §

What survives the six experiments:

1. Structure score as a method router. Useful if you’re already running multiple scoring strategies and want to pick per task. +2-6pp over global strategy. Not a standalone product.

2. Negative results worth writing down. “Divergence doesn’t generalize” is a thing a lot of people assume without checking. “Structure score is a method router, not a variance router” is a mistake I almost made and several people I talked to independently proposed. Documenting them saves someone else three weeks.

3. The discipline of held-out validation. The in-sample results looked great. The held-out results killed most of the pipeline.

4. moirai itself as a per-task debugging tool. If you have traces and want to know where your runs diverge and which path worked, it works. On SWE-rebench v2, 77/1096 tasks surface strong claims like “wrote source → 85% success (13 runs) vs ran a command first → 0% success (3 runs)”. It’s a personal tool, not a product.

Hard lessons from building this §

Same section as the last post. These are the things I wish someone had told me at the start.

Prove signal before building infrastructure §

My original plan was SFT → ORM → DPO with divergence-mined pairs, running on OpenHands eval infrastructure. Three stages of training, plus eval harness work. Six to eight weeks of build before the first result.

My advisor pushed back: “prove the signal first.” I compressed the plan into two validation experiments, held-out prediction and reranking, and ran them in three days. The signal was weak. The entire training pipeline plan evaporated in 72 hours.

If I had skipped that step, I would have built the training loop, run it, gotten noisy results, and then spent weeks debugging “training instability” that was actually a signal-strength problem. Held-out validation is cheap. Training loops are not. Run the cheap experiment first.

Family-conditional signals look universal in aggregate §

The first round of analysis on eval-harness data showed strong behavioral patterns: edit-test alternation and recurring motif sequences. Then I stratified by task family. Everything disappeared. The “universal” patterns were family artifacts. Certain families have certain shapes, and when those families dominate the dataset the pattern looks global.

This is Simpson’s paradox in agent eval, and it’s everywhere. Always stratify. If you can’t stratify because your dataset is too small per family, you don’t have enough data to draw the conclusion you want.

Don’t reuse one latent variable for two jobs §

I had structure score working for method routing. I assumed it would also work for eval budget allocation. Both are “use this variable to make a per-task decision,” same shape. Different physics.

Structure measures branching sensitivity. Volatility measures outcome uncertainty. On SWE-rebench they’re anti-correlated. Test each use of a variable on its own terms before assuming it transfers.

Don’t overclaim §

The pull to frame a +2-6pp routing rule as a “control system for stochastic agents” or a “reasoning paradigm” is real, and it’s killed every time I tested it against a serious reader. A routing rule is a routing rule. Name it that. The honest framing is easier to defend and easier to iterate on.

Aggregate metrics hide the regimes §

The headline number for divergence-based reranking was +3.2pp. The per-family captured oracle gap ranged from -73% to +100%. The aggregate hid two different problems on two subsets of the data. Once I saw the per-family distribution, everything else followed.

Any time you’re computing an aggregate metric over a heterogeneous dataset, look at the distribution. The mean is almost never the story.

The point §

Agent trace divergence contains signal. The signal is weaker and more task-specific than the in-sample numbers suggest. On held-out data, raw divergence is a coin flip. Behavioral features recover a modest amount. Routing between scoring strategies by structure buys you another two points.

None of that justifies a preference-learning pipeline. SWE-bench teams have converged on SFT for a reason, and the reason is consistent with what I found.

If you’re thinking about extracting training signal from multi-run agent traces, run the cheap experiments first. Held-out AUROC. Reranking lift. Per-family captured oracle gap. If those numbers are weak, and they probably will be, the expensive infrastructure won’t save you.

Use the signal where it’s real: method selection on the minority of tasks where the divergence is early and decisive. Treat everything else as drift.

Try it yourself §

The code for all six experiments is public:

github.com/orban/moirai: Python, ~7k lines, 423 tests.

git clone https://github.com/orban/moirai.git
cd moirai
pip install -e .

# Run branch analysis on your own traces
moirai branch path/to/traces/ --html report.html --json report.json

# Reproduce the held-out study
moirai holdout path/to/traces/

# Reproduce the reranking experiment
moirai rerank path/to/traces/ --k 3

# Reproduce the structure score routing
python -m scripts.run_structure_routing path/to/traces/

The last script is locked with frozen parameters and an assertion on the output, so you can verify the headline number (59.2% conditional accuracy on SWE-rebench v2) hasn’t drifted.

If you run this on your own multi-run data and find something I missed, I want to hear about it.

Appendix: experimental details §

Data §

• Dataset: SWE-rebench v2 from Nebius
• Scale: 1,096 tasks with mixed-outcome runs, 12,854 total runs, median 11 runs/task
• Filter: tasks with ≥3 runs and both pass and fail outcomes
• Alignment: Needleman-Wunsch at step-name granularity (e.g., read(source), edit(test_file))

Holdout study §

For each task with ≥6 runs, randomly split runs 50/50 into train/test (stratified by outcome). Find divergence points on train with min_branch_size=2, q_threshold=0.5. Score each test run by the signed sum of its step contributions at known divergence columns. Measure per-task AUROC via Mann-Whitney U. Aggregate as pooled AUROC across all test runs and as per-task mean AUROC.

Reported results are over 1 seed. Spot-checked with 3 seeds and deltas were within 0.01.

Reranking study §

For each task with ≥3 runs, sample K=3 runs without replacement. Score each with the method being evaluated. Pick the top-scored run. Record whether it passes. Repeat 500 times per task. Aggregate accuracy across tasks.

Confidence intervals via Wilson score. Per-family captured oracle gap = (method_accuracy − random_accuracy) / (oracle_accuracy − random_accuracy).

Structure score §

For task t:

• branch_gap = mean outcome spread across top-5 divergence points, range [0, 1]
• earlyness = 1 − mean position of top divergence points (normalized to trajectory length)
• stability = fraction of 20 subsamples (at 70% of runs) that recover at least one top-5 divergence column within 10% position tolerance

Threshold of 0.20 for “high structure” was picked by inspecting the bimodal distribution on a held-out slice, not optimized on the test set.

Eval reliability simulation §

For each task with ≥6 runs, treat the full run set as ground-truth pass rate. Simulate 2,000 evals per policy by sampling k runs with replacement from the ground-truth outcomes. Measure absolute error of estimated rate vs ground truth and false signal rate between pairs of independent simulated evals (disagreement > 3pp).

Budgets tested: 1.0, 1.5, 2.0, 3.0 runs per task. Structure-aware policy spends 1 run on low-structure tasks and distributes the rest uniformly over high-structure tasks.

Code §

All experiments are in moirai/analyze/ with a run_* script per experiment in scripts/. Everything is deterministic under fixed seed. Reproducibility assertions are in scripts/run_structure_routing.py.