That’s a sensible default for a browser suite. Now say a product bug makes one test fail one run in five.
With two retries, the test fails all three attempts only 0.2 × 0.2 × 0.2 = 0.8% of the time, so the build stays green in 99.2% of runs. I planted such a bug in a small service and ran its test suite 41,200 times.
Retries hid the bug about as often as the arithmetic predicts, while the test reports flagged the bug’s test as flaky in about one green build in five. Below are the numbers and the quarantine policy I’d build on them: Who owns a quarantined test, how long it may stay and how it gets out.
Test Tools Flag a Pass on Retry, But the Exit Code Doesn’t The tools I checked do record a pass on retry. Playwright labels the test ‘flaky’, pytest-rerunfailures prints an R for each rerun and Maven Surefire counts it under ‘flakes’. Still, the exit code is still 0, and CI passes or fails the job on the exit code.
