A simulation environment, termed vera-gym, is part of a suite of tools developed by Vera Platforms, a startup originating from the University of Michigan research group, designed to make AI agents reliable enough for DevOps responsibilities. The deployment and operation of software, the focus of DevOps, is inherently a high-risk domain. While AI agents possess a degree of latitude to make and correct errors during code generation or prompt processing, the situation changes drastically when they modify live infrastructure supporting operational workloads.
"Live infrastructure is unforgiving," stated Ang Chen, who serves as the CEO of Vera Platforms and is an associate professor at the University of Michigan, in an interview with The Register. He elaborated that an erroneous modification could result in service failure, the creation of a security vulnerability, or data loss. Furthermore, mistakes often cannot be undone, meaning there is no safe avenue for experimentation; running code against actual production infrastructure simply is not a viable option.
Over several years, Chen and his collaborators have been developing a method to evaluate agentic DevOps capabilities, which includes the creation of a simulation environment, or "gym," intended to improve AI models' proficiency in DevOps functions. Chen noted, "From the research that we did, we've figured out various weaknesses of leading models." He pointed out that even though [Anthropic's] Claude is very proficient at generating Python code, it possesses certain inherent limitations that prevent it from succeeding in DevOps scenarios.
As an illustration of this limitation, Chen mentioned that when both Claude and OpenAI's Codex were tasked with removing resources within the AWS cloud, these models failed to consider regions outside of the default setting. Chen clarified, "The default region is us-east-1." Since AWS comprises numerous regions, Claude lacked the comprehension that cloud resources could exist across multiple geographical areas, leading it to only remove a portion of the intended resource. The root cause, he explained, stems from the fact that these models have processed substantial amounts of public code but have encountered significantly less operational tracing data.
Vera Platforms intends to address these deficiencies through three distinct offerings: vera-gym, which functions as a sandbox replicating production infrastructure; vera-bench, a metric for assessing where AI agents falter in DevOps tasks; and vera-agents, AI agents specifically trained within vera-gym and prepared for deployment in DevOps roles. The CloudOpsBench test, which is forthcoming, is an evolution of work presented at NeurIPS in 2024. Chen reported that Claude and Codex currently achieve only about a 50 percent success rate on this assessment. He characterized this as, "It's a very, very hard set of tasks."
This benchmark encompasses responsibilities such as deleting retired S3 buckets, rectifying a pipeline related to compliance events, and restoring DynamoDB write autoscaling. Chen added that "The tasks are authored by senior infra operators with many years of experience, with rigorous peer review." A significant benefit of this benchmark is that it does not necessitate an actual cloud deployment. Chen stated, "We have a high fidelity simulator that mimics Amazon behavior," adding, "So the rollout, the performance of the agent on the real cloud and the performance of the agent in our emulator is exactly the same. So for every API, we have this comparison and the agent sees the same thing." The availability of vera-agents is projected for Q1 2027. In their academic investigations, the researchers have described cloud infrastructure work as "critical but tedious work." With further refinement through training, it is conceivable that AI agents might assume these monotonous duties.
The development of robust simulation and benchmarking tools underscores a broader industry movement toward automating complex operational tasks using artificial intelligence. The efforts by academic researchers and startups like Vera Platforms reflect the growing necessity among infrastructure operators to move beyond rudimentary prompt-based interactions to ensure the safety and reliability of automated systems interacting with production environments.
Why it matters
For infrastructure operators, deploying AI agents into production environments carries significant risk of irreversible service outages or security holes. This development addresses the critical need for reliable, safe testing environments before trusting AI with live cloud resources.
