Agree
Take approved, anonymized traces from three teams. Build a small jointly approved test set.
Evaluation example / methodology, not customer proof
Keep the denominator.
Keep the uncertainty.
Eight seeded fixture runs show a controlled example of task variation. They do not establish production reliability or customer impact.
Exactly-one-draft criterion
Change the condition.
Inspect the run set.
8 / 8Baseline met6 / 8Candidate met
No sampled confidence interval is claimed for these deliberately constructed fixtures.
DEMO — SYNTHETIC DATA Simulation only
✓1011 draft
✓1021 draft
✓1031 draft
×1042 drafts
✓1051 draft
✓1061 draft
✓1071 draft
×1082 drafts
Candidate v0.9 / fixed synthetic seeds / timeout
Duration ≠ task success
One measure
cannot tell the whole story.
The dot plot reports illustrative duration while color and text identify duplicate artifacts. The task criterion remains independent.
Read duration data
| Seed | Seconds | Drafts / timeout |
|---|---|---|
| 101 | 82 | 1 |
| 102 | 91 | 1 |
| 103 | 87 | 1 |
| 104 | 118 | 2 |
| 105 | 96 | 1 |
| 106 | 103 | 1 |
| 107 | 89 | 1 |
| 108 | 132 | 2 |
Read diagram description
8 synthetic durations, 82–132 seconds; seeds 104 and 108 create duplicates
Proposed buyer experiment
Evidence that earns
the next step.
Compare
Blind-compare the readout against each team’s existing LangSmith, Braintrust, or internal gate.
Commit
Advance with an independently confirmed missed regression and an engineering lead’s funded trial or real gate commitment.
A smaller test. A clearer decision.
Start with the workflow
you can’t afford to guess.
Agree on one task criterion, an approved fixture, and a named engineering owner.
Request a replay study