Project Zero conducted a comparison between two automated tools utilized by Google to detect flaws within Chrome's V8 JavaScript engine.
The tools examined were Fuzzilli, which is a generative JavaScript engine fuzzer, and BigSleep, an LLM-based bug finding system. Over the preceding six months, BigSleep identified more vulnerabilities than Fuzzilli, reporting 45 versus 37, with a 25% overlap in the specific issues reported.
These findings were supplemented by dedicated experiments designed to test theoretical overlap in vulnerabilities found since 2023, which estimated the overlap to be approximately 60%. Despite this variance, the analysis indicated that a significant quantity of vulnerabilities could be discovered by either LLMs or by fuzzing alone.
The researchers concluded that while the rapid discovery of flaws by LLMs is highly encouraging, fuzzing and LLMs remain complementary methodologies. Vendors are advised to employ both fuzzing and agentic analysis on their code to maximize vulnerability reporting prior to user impact.
Fuzzing has a history of successful application to software like Chrome and its V8 engine for bug detection. Recently, the advent of powerful LLMs capable of reading, reasoning about, and writing code is introducing a transformative shift into software security and development.
To address the question of whether LLMs are replacing or complementing traditional fuzzers, the study focused on two production systems: Fuzzilli and BigSleep. Both automatically scan newly added code, allowing for a direct comparison.
Fuzzilli operates continuously across multiple instances, typically identifying new changes within hours, whereas BigSleep currently runs once daily, prompting reviews of new commits to generate proof-of-concept testcases for discovered bugs. Both systems submit findings via Clusterfuzz for initial crash triaging before filing reports in Chrome's issue tracker.
The analysis focused on V8, a mature, high-performance JavaScript and WebAssembly engine, as Fuzzilli is a specialized tool for these engines. The study queried Chromium’s bugtracker for the six-month period from April to September 2026, as the BigSleep pipeline commenced in April of that year.
When filtering for only 'vulnerability' type bugs, the comparison showed a near tie, with a slight lead for BigSleep (45 versus 37). Further refinement by focusing only on "actionable" bugs—those not marked as duplicates or WAI and possessing at least one referencing fix commit—yielded a final vulnerability count of roughly 45 versus 37.
Examining overlap, the initial bugtracker review showed an overlap of about 25% (20 out of 81 actionable reports), with Fuzzilli often finding the bug first. A subsequent experiment attempting to rediscover historical Fuzzilli bugs using BigSleep yielded 112 final, deduplicated challenges, demonstrating the complexity of cross-system validation.
This investigation highlights that the integration of advanced reasoning models alongside established fuzzing techniques is crucial for comprehensive security coverage within platform infrastructure.
