0
Root-cause a flaky test using a structured hypothesis funnel
⁂auto-checked, 4 days oldAauraNovice
The prompt
This test fails intermittently, not on every run. Help me find the actual root cause instead of just adding a retry or a sleep. Work through this funnel and show your reasoning at each step:
1. List every source of non-determinism visible in the test and the code it exercises (ordering, timing, shared state, randomness, network, concurrency, uninitialized globals).
2. Rank them by how likely each is to explain the specific failure signature I give you.
3. For your top hypothesis, propose the smallest possible experiment (a log line, an assertion, a seed) that would confirm or rule it out, without changing behavior.
4. Only after that, propose a fix. Explicitly say why a sleep/retry would mask this rather than fix it, if that's what's tempting.
5. If the fix requires changing production code (not just the test), flag that clearly since it has broader blast radius.
Test code: test_code
Code under test: code_under_test
Failure signature (error, stack trace, how often it fails): failure_signature
Where it has been run
| Model | Version | People | Broke | Auto | Last |
|---|---|---|---|---|---|
| Claude | Haiku 4.5 | 0 | 0 | 1/1 | 4 days ago |
Example output
Non-determinism sources: (1) test iterates a dict and asserts order, (2) shared in-memory cache not reset between tests, (3) a background thread flushes every 50ms which can race the assertion.
Most likely: #2, because the failure only appears when this test runs after `test_cache_eviction` in the suite, and never in isolation, matching a shared-state signature rather than a timing one.
Experiment: add a `print(id(cache_instance))` at test start in both tests; if the ids match across tests in the same run, that confirms the cache singleton is leaking state between tests.
Fix (pending confirmation): add a fixture that resets the cache singleton before each test, not a `time.sleep`. A sleep would hide the leak and the next engineer who adds a slower op nearby would reintroduce the flake. This only touches test setup, no production code change needed.
0 comments
Sign in to comment or report what this prompt did for you.
Sign inNo comments yet. Run the prompt and report what you got.