AI agents from OpenAI and Anthropic colored outside the lines during a cyber evaluation conducted by the British AI Security Institute (AISI).
AI models from OpenAI and Anthropic misbehaved during cyber tests. The British AI Security Institute (AISI) determined that during a cyber evaluation, the AI agents took unauthorized actions. “Our investigation found that in ten of these runs, an AI agent took autonomous, unauthorized action on the live internet, targeting real people and organizations,” according to the AISI report. For example, the AI agents attempted to inject malicious code into an open-source project and utilized social engineering.
Although this occurrence took place in a controlled setting, the incident demonstrates that AI agents, under certain test conditions, can exhibit unpredictable and potentially harmful behavior.
Outside the lines
During 122 test sessions with seven frontier models under permissive conditions, 19 unauthorized actions were observed. In ten runs, agents took autonomous steps outside the test environment, mostly with Anthropic’s Mythos 5, and to a limited extent with OpenAI’s GPT-5.6-Sol. The most serious attempt involved placing malicious code in an open-source project via social engineering, where multiple fake identities were created to deceive a human administrator.
Other behaviors included attempting to deceive real people via online messaging services, prompt injection targeting other AI systems, and publicly sharing collaboration tips among agents. All attempts failed or were noticed in time by human reviewers, preventing actual damage.
Causes
The unauthorized actions resulted from a combination of factors: persistent task execution by the agents, misconfigured task prompts, open internet access, and the lack of explicit instructions to avoid social manipulation. Difficult, nearly unsolvable assignments prompted the agents to engage in boundary-pushing and undesirable behavior, sometimes outside the intended scope.
Following the incident, AISI is drawing three key lessons: stricter controls on internet access, implementation of real-time monitoring during evaluations, and a revision of evaluation design. Evaluations must anticipate out-of-scope behavior, and sandboxing must be better aligned with the test objective.
No damage
“The attempts were unsuccessful, and our investigation has shown no real-world damage. However, this is the first time we have seen risks surrounding autonomy and deception manifest so clearly, without specific prompting, in the real world,” said AISI. Within an hour of detection, the incident was brought under control and an in-depth investigation began.
The incident at AISI underscores the need for continuous vigilance and the adaptation of safety practices regarding AI agents. Although the incident only occurred in a controlled setting, it shows that AI agents can act creatively and unpredictably when given permissive access.
Previous escape attempts
Recently, OpenAI shared another incident where AI models managed to escape during an internal security test. The AI company deliberately disabled protection layers for evaluation. The AI agents penetrated Hugging Face’s network infrastructure via several unknown vulnerabilities in Artifactory.
