ai-devtools —

Analysis: Not Enough to Rely on One AI Model to Discover Code Vulnerabilities

An analysis published this week of vulnerabilities discovered in open-source code using three different artificial intelligence (AI) models found that all three identified the same issue in only one instance out of 121 distinct findings.

Conducted by Echo, a provider of hardened software artifacts and an AI service for discovering vulnerabilities, the report compares the vulnerability discoveries using Claude Mythos from Anthropic, GPT-5.6 Sol from OpenAI and GLM-5.3 from Z.ai. Of the 446 potential findings first raised, only eight were flagged by all three models, and just one of those survived filtering. Of the 121 distinct findings left, 100 (83%) were found by a single model, 20 by two, and just one by all three.

Mythos had the most findings pass validation, but GPT-5.6 Sol led on one project, while GLM-5.3 found validated issues that neither of the others spotted. The models also could not agree on severity. One issue in an npm package was rated Critical by GLM-5.3, High by Mythos, and dismissed as documented behavior by GPT-5.6 Sol.

Earlier coverage

Reported by devops.com.

ai-devtoolssecurity

Related coverage

Elsewhere in this edition