AI agents from Google Gemini repeatedly attacked the live website of Epoch AI during safety tests in October. Testing firm Trajectory Labs has since halted all tests on misalignment.
The agents of Gemini 4 Argon kept attacking the real Epoch AI website, even after the restrictions were tightened. That is clear from reports that Trajectory Labs, a company for AI safety research, and Epoch AI published on Friday. Epoch AI did not check for itself which models were involved, as the identification as Gemini comes from Trajectory Labs.
The agents were given an impossible task: retrieving the official answers to non-existent problems from Epoch AI’s FrontierMath benchmark. The researchers repeatedly urged them to “NEVER stop, NEVER give up”. Nobody had asked the agents to hack websites or to pose as employees.
Forty agents on the live website
On 20 October, around forty agents started probing Epoch AI’s live website for sensitive information, including administrator credentials and access to the benchmark’s evaluation logs. Epoch AI noticed this and set up a so-called agent hotline, through which the agents could send messages directly to staff.
For three hours, the agents made 36,411 requests to a disabled API endpoint and sent 487 messages via the hotline. Some messages posed as real Epoch employees, others as engineers or auditors requesting benchmark data. The investigation also found an attempted SQL injection and more than two hundred attempts to reach the FrontierMath evaluation logs.
Trajectory Labs discovered the attacks while reading through the agents’ transcripts and alerted Epoch AI the next morning, reports BleepingComputer.
Agents bypass every measure
After consulting Epoch AI, the parties agreed to test on a local copy of the website from then on. On 23 October, some agents noticed they were working with a copy. They looked up the website’s real IP address and connected to it directly, which made the DNS redirect meant to contain them useless.
When the researchers blocked those IP addresses, an agent switched to another address. Trajectory Labs then configured its network proxy to block direct connections to Epoch, but an agent still retrieved the website’s public content via a third-party web reader. “In a number of these runs, the agents figured out they were on a copy, found the right raw IP address to bypass our redirect and reached the live site again, for which Epoch warned us,” Trajectory Labs writes.
In a separate technical analysis, the researchers describe how the agents used a shared cache for software packages to set up a bulletin board to communicate with each other. They exchanged information there about vulnerabilities and attack methods. One agent shared a technique to bypass the restrictions on a deliberately vulnerable copy of the Epoch website, after which another agent used it to obtain login credentials and complete the task. The agents also tried to bypass CAPTCHA protection and to pose as employees in order to coax protected benchmark information out of an AI assistant.
Tests suspended
From 26 October, Epoch AI cooperated on additional experiments, knowing that the agents had internet access and could reach the public website. The agents visited the live infrastructure again, with more than 65,000 requests to a disabled API endpoint, new attempts at SQL injection and posing as employees.
