Replay methodology / 3 min read

A seed is not a guarantee.

Repeatable fixtures and deterministic live agents are different promises.

FlakeMap research note / Synthetic example / 8 October 2026

Control what the lab controls

The local preview selects fixed stub responses using an explicit seed. Repeating seed 104 under the timeout condition reproduces the same two-draft fixture.

Name what remains variable

Live model versions, tool responses, changing data, and timing can introduce variation outside a fixture. A seed by itself does not remove those sources.

Report the denominator

Show the run set and condition. Eight fixed illustrative runs are not a production reliability estimate, and they do not justify a confidence claim.

Order lookupPO-204

Stock checkTimeout injected

PO draftRetry + late response

↳Two paths reach draft creation.Candidate lacks a shared deduplication guard.

Inspect the example