Find the cause of a flaky test or intermittent bug

Systematically narrow down why a test or bug only fails sometimes, with experiments to prove the cause.

LLazyPrompt(@lazyprompt)CodingAnalyzeWorks with any model
No ratings yetSign in to rate
Save 0 Fork Report
Prompt template
I have an intermittent failure in {{language}} and need a systematic way to pin it down.

What fails and how often: {{frequency}}
Where it fails (CI only, locally, specific machines): {{where}}
Failure output from a failing run:
{{failure}}

Test or code involved:
{{code}}

What changed recently, or other clues: {{clues}}

Please:
1. List the most likely categories of cause for this specific situation, ranked, from this set: shared state between tests, ordering dependence, timing and race conditions, unawaited async work, real clocks and time zones, randomness, resource limits, environment differences, and external services. For each, say what in my information points to it.
2. For the top three, give an experiment that would reproduce or rule it out. Be concrete: a command, a loop to run the test 200 times, a flag to randomize or fix order, a log line or a delay injected to widen a race window.
3. Explain how to tell which result means what, so I can read the experiment outcomes.
4. Once the cause is found, outline the right kind of fix for each category. Warn me against the tempting wrong fixes such as sleeps, retries or increased timeouts.
5. Suggest instrumentation to leave in place so a future occurrence leaves a useful trail.
Example output
**Likely causes (ranked)** 1. Shared state between tests (high). It began when parallel workers were enabled, and the failure is "expected 1 row, got 3". Two workers likely write to the same table. 2. Race with unawaited async work (medium). `createUser` is called without `await` in the setup helper, so the assertion can run first. 3. Environment resource limits (low). CI has 2 cores; slower timing could expose a race. **Experiments** 1. Run `vitest --no-file-parallelism` 30 times. If it never fails, shared state is confirmed. 2. Add `await` to the setup call, then loop: `for i in $(seq 200); do vitest run users.test.ts || break; done`. 3. Run under CPU throttling: `taskset -c 0 vitest run`. **Reading results:** fails only in parallel means isolation; fails alone in a loop means a race inside the test. **Right fixes:** per-worker schema or transaction rollback for isolation; awaiting the promise. Avoid adding sleeps or `retry: 3`, which hide the defect. **Instrumentation:** log worker id and database name at test start.

Why it works

Ranking cause categories against your clues, then pairing each with an experiment, turns guesswork into a process. The warning against sleeps and retries steers toward real fixes.

Licensed CC0 (public domain) Version 1 · history