The complexity of measuring AI model security is rapidly increasing due to a proliferation of benchmarks, each targeting different variables. However, these external tests cannot substitute for evaluation against an organization's specific code and operational workflows.
Cybersecurity vendors and developers of foundational models are enhancing the performance and efficiency of their AI systems. Quantifying these performance gains presents a challenge, especially when claims of models with smaller token costs arise weekly. Determining if these models match the capabilities of more general-purpose members within a model's family remains difficult. Organizations are advised to test models using their own code and workflows to align with business needs.
For instance, the national sovereign AI and cyber-defense firm Dream Group trained an 8 billion parameter model, one of the smallest modern models, using 220,000 vulnerability records, threat intelligence reports, detection rules, and other cybersecurity documentation. This model, named the Mid-trained Security Transformer (MiST), boosted average performance across 13 cybersecurity tasks, primarily involving multiple-choice and true-false questions related to vulnerability analysis and cybersecurity domains.
Elad Ben Zaken, an AI researcher at Dream Group, stated that improved security capabilities do not necessitate a more complex model. He noted that specialized training and augmenting the model with superior software both enhance outcomes. He commented, "I expect efficiency to keep improving in coding and security tasks through specialization and better workflows," adding, "The model supplies capability; the harness determines how effectively it uses context and tools. Adding more of either can increase cost without improving results."
While Anthropic reported that its Opus 5.5 improved performance while costing 40% less than Opus 5 on "typical workloads" when released in September, this is not representative of cybersecurity tasks. A model's performance on one task may hold little value for enterprises seeking expert systems for secure code generation or for mapping enterprise attack surfaces.
On the offensive front, researchers are developing bespoke benchmarks for cybersecurity-focused models. Albert Ziegler, head of AI at offensive-security firm XBOW, mentioned that metrics for this firm concern the capacity to discover and exploit weaknesses. Because different clients require specific models, XBOW develops its own harness to optimize results regardless of the model employed. Ziegler stated, "In its most atomic form, that's the ability of the model to find an exploit — that is our bread and butter in terms of seeing what's the value that the model produces," adding, "For other companies, this bread and butter could obviously be something else."
The SEC-Bench Pro, created by a research team at the University of Illinois Urbana-Champaign, attempts to measure an AI agent's ability to independently locate and trigger a known memory-safety bug in a browser engine or the Linux kernel, provided the general location and bug type are specified. This metric ranks Codex and GPT-5.5 as current leaders, achieving an overall solve rate of 58%, while noting that open-weight models generally struggle, with GLM-5 succeeding in only 13 of 344 test cases.
Booz Allen, a government contractor, has published the Cyber Weapons Index (CWI), which combines a model's ability to find and exploit vulnerabilities with its effectiveness in executing attacks against a target. The latest iteration places OpenAI's GPT-6 Astra at the top with a CWI of 80.5, closely followed by Claude Mythos with a score of 80. Eric Syphard, senior vice president and head of AI at Booz Allen, stated, "The CWI offers one empirical approach to offensive cyber capability — test models against a common enterprise network and score demonstrated actions using network evidence, rather than relying on what models say they can do," and added, "There is no single measure that captures every security risk. Evaluations need to be designed around the capability being assessed and should make their methods, environment, and success criteria clear."
Dream researchers also found that expert curation of a primary knowledge database, termed a "seed corpus," can generate domain-specific synthetic training data, thus lessening dependence on general knowledge databases for specialized models. The MiST-8B and MiST-30B-A3B models were evaluated across 13 test datasets, including 1,072 multiple-choice questions about MITRE ATT&CK (SECURE-MAET) and 870 true-false questions regarding code vulnerability (the PrimeVul P-C).
While offensive metrics and defensive capability measurements are advancing rapidly, assessing areas like safety proves more challenging. The UK's AI Security Institute (AISI) has been benchmarking AI models on completing a 32-step attack chain, with Anthropic's Claude Mythos Preview and OpenAI GPT 5.5 currently capable of this. Joe Tyler, general manager and AI researcher at Sonar, noted that newer models do not inherently imply greater security or advanced security capabilities. He advises that while public benchmarks can help narrow down candidates, organizations must conduct evaluations reflecting their specific code and workflows, stating, "Start with the task and the failures you can't accept, rather than assuming the newest model is the right choice."
The rapid evolution of AI adoption means that enterprises must prepare for an increased volume of vulnerabilities and quicker exploitation timelines, according to Nico Waisman, CISO at XBOW. Waisman recommends that organizations invest in processes and tools that accelerate security teams, automate routine tasks, and shorten remediation periods, emphasizing, "Ultimately, I wouldn't evaluate AI security tools based on which model they use or how impressive their demos look," but rather on "measurable outcomes: Are they helping you identify real risks? Are they reducing your exposure? Are they making your team significantly faster? That's where the security budget should go."
These diverse evaluation methodologies reflect a broader industry trend where the technical improvements in AI systems, such as those seen with Anthropic's Opus 5.5, must be contextualized against the specific operational risk profiles of various market participants.
