'Replacing humans is not close': BlockSec challenges EVMBench on AI auditing

Researchers at BlockSec have found that EVMBench, the artificial intelligence-powered smart contract auditing benchmark developed by OpenAI and Paradigm, may have been overly optimistic about AI's ability to automate away human oversight. 

EVMBench tested AI agents on smart contract security tasks like detecting, patching, and exploiting vulnerabilities, with reportedly impressive results.

In a blog in February, EVMBench devs said AI was able to exploit 72% and detect about 45% of smart contract vulnerabilities, using 120 curated examples from Code4rena audits, suggesting "discovery" was the main bottleneck in automated auditing.

BlockSec wanted to retest these results, noting in a recent paper titled “Re-Evaluating EVMBench” that OpenAI and Paradigm’s testing conditions may have confounded its results. 

“EVMBench says AI can exploit 72% of smart contract vulnerabilities, and the industry started talking about fully automated auditing. We re-tested with more configurations and 22 real-world attack incidents. Exploit success: 0%,” BlockSec co-founder Yajin Zhou said in an X post.

The authors boosted the number of model configurations to 26 by mixing and matching different bots to different model scaffolds, meaning, for example, running Claude on ChatGPT’s architecture, arguing that the original testing of 14 agent configurations largely confined models to their native vendor scaffold. 

“You cannot tell whether an agent's performance reflects the model's capability or the scaffold's advantage,” they argued. 

Further, BlockSec raised concerns about data contamination in the original report, which tested known vulnerabilities previously published in 40 Code4rena repositories, which could have ended up in an AI’s training data. 

To get around this, the authors tested bots on 22 real-world security incidents that all occurred after mid-February 2026, “so these incidents fall outside every model's training window.”

The results

Perhaps most significantly, the authors found that of the 110 agent-incident pairs they tested, involving five agents running the same 22 incidents, zero end-to-end exploits succeeded, suggesting that even state-of-the-art AIs are far from running real-world exploits. 

That said, BlockSec’s ReEVMBench test results for AI vulnerability detection were largely in line with the original report, with Claude Opus 4.6 performing the best by catching 13 out of 20 real-world vulnerabilities.

“The difficulty distribution follows a clear pattern. Six incidents were detected by nearly all agents (87.5% to 100%), involving well-known patterns like sell-hook reserve manipulation and unchecked multiplication overflow. But four incidents were detected by none, and five by only one of eight agents,” Zhou wrote. 

“These findings challenge the narrative that fully automated AI auditing is imminent. Agents reliably catch well-known patterns and respond strongly to human-provided context, but cannot replace human judgment,” he added.

Zhou concludes by noting that "EVMBench is a valuable contribution" by providing an evaluation standard for the crypto security industry. He also said that AI and human researchers are already performing different, equally useful tasks, each complementing the other's weaknesses.

"The real question is not 'can AI replace humans?' but 'how should humans and AI work together?' AI handles breadth (systematic scanning); humans handle depth (protocol knowledge, adversarial reasoning). Neither can do the other's job. Together, they form a complete audit capability," Zhou wrote.

"AI auditing has real value, but replacing humans is not close. The right direction is human-AI collaboration," he added.