Find the failing step in long-horizon agent workflows before release. Seeded replay, controlled tool faults, and human-reviewed evidence.
For AI engineering leads and evaluation teams shipping multi-step agents.
Why single-run passes can miss it ↗01 / One passing run
A fictional purchase order travels through lookup, stock check, and drafting. The baseline produces exactly one draft.
Acceptance criterion
Order lookupPO-204Stock checkResponse received
PO draftExactly one artifactPurchase orderADEMO — SYNTHETIC DATA / Baseline v0.8
02 / Change one condition
The retry completes. Then the late original response completes, too. In the candidate, both paths create a draft.
Failing transition / duplicate artifact
Injected stock timeoutSeed 104 / candidate v0.9Purchase orderAPurchase orderBDEMO — SYNTHETIC DATA / Candidate misses shared deduplication
03 / Show the distribution
Keep baseline, candidate, tool condition, and seed set aligned. Under the timeout fixture, two of eight candidate runs create duplicates.
DEMO — SYNTHETIC DATA / Seeds 101–108
04 / Keep human judgment in view
The reviewer links the two artifacts to the exactly-one-draft criterion. A proposed gate carries that rationale forward.
Inspect and annotate
Mock gate blocked“Both paths create an artifact. The task criterion fails. Block the mock candidate gate.”
DEMO — SYNTHETIC DATA / No real CI gate changed
02 / Five proposed modules. One evidence chain.
From a consented fixture to a reviewed release decision. Each module is a separate product direction to test, with the first pilot scoped to one.
{
"order": "PO-204",
"tool": "stock.lookup",
"source": "redacted",
"expected_drafts": 1
}Scenario readyExactly one draftFrom approved traces to explicit test fixtures.
Seeded runs with a visible denominator.
Failing transition: second draft creation
Timeouts, delays, and stale tool responses.
Mock gate blockedAcceptance criteria connected to human review.
Failure clusters, evidence, and adjudication.
03 / Isolate the transition
Follow the tool interruption through event order and artifact creation. A passing response does not explain a failed task.
Explore Trust ProofFailing transition: second draft creation
Hover to preview. Click or tap to hold. Escape to return.
DEMO — SYNTHETIC DATA
04 / The evidence workbench
Choose a version, condition, and seed. Compare artifacts, record a mock review, and prepare a synthetic gate report.
Harbor Parts / PO-204Replay workbenchSeeded run matrix
| Seed | Baseline | Candidate | Inspect |
|---|---|---|---|
| 101 | 1 draft | 1 draft | |
| 102 | 1 draft | 1 draft | |
| 103 | 1 draft | 1 draft | |
| 104 | 1 draft | 2 drafts | |
| 105 | 1 draft | 1 draft | |
| 106 | 1 draft | 1 draft | |
| 107 | 1 draft | 1 draft | |
| 108 | 1 draft | 2 drafts |
candidate / seed 104
Criterion failedThe retry and late original response both reach draft creation. The candidate misses the exactly-one-draft criterion.
Purchase orderAPurchase orderB
Source: fixture/po-204.json · timeout · simulation only
Human adjudication / mock state
An engineer reviews the artifact and records why the criterion is met or failed.
PO-204 is fictional. Seeds 101–108 select fixed stub artifacts. Candidate timeout fails on 104 and 108; delay fails on 108; normal meets 8/8. Baseline meets 8/8. Drafts A and B denote artifact origins, not real purchase orders. Owner and review entries are local mock state. All external sends, billing, tool writes, and CI actions are absent.
05 / Existing-stack fit
LangSmith and Braintrust already support evaluation workflows. FlakeMap’s proposed study tests incremental value in long-horizon failure analysis.
See the proposed architecture →LangSmithDatasets, evaluators, trace contextFlakeMap studyFault condition → artifact → reviewerProposed relationship; no live integration or endorsement.
06 / Validation before expansion
Start with a small, jointly approved replay set. Compare the findings with your current gate. Advance when a missed regression is independently confirmed and an engineering lead commits.
Explore the study modelScope and pricing are agreed before a study. No active checkout.
Request a replay study ↗Reliability, with the denominator
Compare one workflow across the same eight seeds. Change the condition to see exactly where the candidate diverges.
Runs meet the task criterion
Runs meet the task criterion
Runs create a duplicate draft
Retry + late original responseCustomer stories / An open invitation
Bring a release question worth answering. Document the failing transition, compare the existing gate, and decide what the evidence changes.
Build a replay study with usCustomer stories will appear here with participant approval.Before you take the next step
The public workspace is an isolated synthetic demo. It replays fixed fixtures without connecting to production tools or CI. Contact us to discuss a scoped study.
The proposed study layers a specific failure-analysis question onto your current process: does a controlled tool interruption change the final task artifact across repeated runs? Incremental value remains to be validated.
No. Public forms do not accept traces or sensitive records. Any eventual fixture sharing needs explicit approval and a separate scoped process.
A meaningful regression missed by the current gate is independently confirmed, and an engineering lead funds a trial or commits a release gate with owned success criteria.
A smaller test. A clearer decision.
Agree on one task criterion, an approved fixture, and a named engineering owner.
Request a replay study