Artificial intelligence models face rigorous evaluations regarding autonomous offensive cyber capabilities. Recently, researchers transitioned from testing isolated tasks to evaluating complete multi-stage attack chains. Consequently, Moonshot AI’s new model, Kimi K3, underwent detailed scrutiny. The UK AI Safety Institute and the US AI Standards and Innovation Center conducted this joint evaluation. Overall, the preliminary findings indicate that Kimi K3 lags behind leading American AI models.
Evaluating ExploitBench: V8 Engine Vulnerabilities
First, evaluators tested Kimi K3 using ExploitBench benchmarks. This specific test required the model to exploit 41 distinct vulnerabilities in Google Chrome’s V8 engine. Kimi K3 achieved a 32% success rate during these trials. Therefore, it outperformed GLM-5.2, which scored 24%.
However, Kimi K3 failed to achieve arbitrary code execution in any test case. In contrast, top American models successfully achieved code execution in roughly 20 of 41 tasks. Thus, a significant capability gap remains between Chinese and American AI models.
Autonomous Network Penetration: The Last Ones Test
Next, researchers placed the model into a simulated corporate network called The Last Ones. This complex environment comprises 32 attack stages, four subnets, and twenty nodes. On average, Kimi K3 successfully navigated to the seventeenth stage. Meanwhile, GLM-5.2 reached only the eleventh stage.
Furthermore, leading American models advanced through an average of 28.5 stages. Nevertheless, Kimi K3 completed the entire attack sequence in one out of ten attempts. Consequently, this outcome demonstrates that Kimi K3 can autonomously exploit poorly secured corporate networks once initial access exists.
Evaluation Limitations and Safety Guardrails
According to a preliminary assessment of Kimi K3’s cyber capabilities, several evaluation constraints existed. The test network lacked active defenses, intrusion detection systems, and real-world security measures. Additionally, researchers pre-staged the vulnerability sequences.
Significantly, Kimi K3’s built-in safety mechanisms failed to prevent exploit generation. Public AI models typically enforce strict safety guardrails during normal operations. In contrast, researchers evaluated American models with safety constraints disabled to measure maximum potential capacity.
Current Landscape of Frontier AI Models
Ultimately, researchers emphasize that these benchmark conclusions remain preliminary. Evaluators assessed Kimi K3 across a restricted set of open and closed tests. Nevertheless, the results provide valuable insights into current frontier model capabilities. Although Kimi K3 surpasses competing open-source models, top closed-source American models retain a decisive advantage.
Support Our Threat Intelligence
If you find our technology report and cybersecurity news helpful, consider supporting our work.
Buy Me a Coffee
PayPal
