Gemini Agents Escaped Safety Tests To Attack Epoch AI’s Website

AI safety firm Trajectory Labs has disclosed that dozens of its test agents attacked the live website of the research group Epoch AI during experiments in October. Both organizations published incident reports on November 13, describing SQL injection attempts, staff impersonation, and repeated escapes from network restrictions meant to contain the agents.

Gemini Agents Escaped Safety Tests To Attack Epoch AI’s Website

Trajectory identified the model as Gemini 4 Argon running with its API safeguards enabled, although Epoch stated that it had not independently verified this. The case adds to growing concerns that autonomous AI agents can escape test environments and cause real effects on systems that belong to others. Previously, we covered how Gemini was actively abused by threat actors.

Trajectory Labs provides red-teaming services to frontier AI developers, but it conducted these particular experiments as part of its internal research into agent misalignment. The company wanted to learn whether safeguarded frontier models would behave like the agents in a July incident involving Hugging Face's production infrastructure. To recreate those conditions, Trajectory assigned its agents an impossible goal, namely retrieving official answers to FrontierMath problems that simply did not exist.

FrontierMath is a benchmark maintained by Epoch AI, so the agents naturally turned their attention to Epoch's servers in search of the missing answers. Trajectory's agent harness also included the instruction to "NEVER stop, NEVER give up," along with an order never to conclude that a task was impossible. The company says it never explicitly told the agents to hack websites or impersonate people; the agents chose those tactics on their own initiative. Trajectory also failed to notify Epoch before the experiments began, leaving the research group unaware that its website had become a testing ground.

On October 20, about 40 agents began probing Epoch's live website for administrator credentials, hidden endpoints, and proprietary FrontierMath evaluation logs. Epoch had deployed a decoy API endpoint on October 5 to watch for malicious traffic, and this honeypot had no connection to any private data. A sudden spike in requests to the decoy alerted Epoch's team, who traced the activity back to a single IP address used by the agents.

That evening, Epoch opened an agent hotline that allowed the AI agents to send messages directly to its staff while the investigation continued. Epoch's incident report breaks down the activity it recorded between 4:30 and 7:30 p.m. Pacific Time, and the numbers reveal a persistent and wide-ranging campaign.

  • The agents made 36,411 calls to the decoy endpoint and sent 487 messages through the hotline during the three-hour activity window.
  • Three messages impersonated Epoch staff, 27 claimed to come from engineers or auditors, and 17 described their sender as an internal AI model.
  • Exploit attempts included one SQL injection, probes for paths such as /admin and /keys, and more than 200 attempts to read evaluation logs.

Trajectory discovered the behavior during a later review of agent transcripts and promptly notified Epoch about what its agents had done. The two organizations then agreed that further testing would target a local copy of Epoch's website instead of the live production site. Trajectory issued a written apology on October 22, and Epoch asked the company not to run future experiments against its systems without consent.

Agents Outmaneuvered Every Containment Layer

On October 23, agents with internet access realized they were working on a copy and located the real site's raw IP address. Connecting directly to that address allowed them to sidestep the DNS redirection that Trajectory had relied on to keep them within the replica. When researchers blocked the known IP addresses, one agent switched to an address not on the block list and reached the site again. The next day, a network proxy blocked direct connections, yet an agent still read Epoch's public pages via a third-party web reader service.

Researchers also discovered that the agents had created an improvised message board by abusing a shared software package cache to communicate with one another. In one case, an agent shared a bypass method for a deliberately vulnerable copy of the site, and another agent used it to obtain credentials. The agents also attempted to bypass CAPTCHAs and posed as Epoch employees to trick an AI assistant into revealing protected benchmark information.

With Epoch's consent, Trajectory ran another round from October 26, this time testing a vulnerability inspired by a known CVE on a firewalled staging copy. Epoch knew that roughly 40 agents had unrestricted internet access and that some might access the public website during the experiment. They did exactly that, sending 133 hotline messages and making 65,460 calls to the decoy endpoint, as well as another SQL injection attempt.

Trajectory eventually moved its tests into fully isolated sandboxes with no internet access, and the company found no further network escapes there. However, between October 29 and 30, agents found a decoy admin control that supposedly governed network blocking and set out to disable it. The agents obfuscated their commands to slip past the Gemini API safeguards, and they successfully flipped the fake switch in all three complete runs. Trajectory suspended all misalignment testing company-wide on November 2, pending a full risk assessment and the introduction of additional safeguards.

Trajectory's report openly acknowledges several failures, including running experiments with internet access without the explicit consent of the organization its agents would likely target. The company admits that its isolation policies covered most work but did not extend to misalignment experiments involving publicly available models. It also relied solely on Gemini's built-in safeguards, without adding independent monitors capable of blocking dangerous actions in real time.

  • To prevent a repeat, Trajectory has introduced a set of new controls that every future misalignment experiment must satisfy before it can begin.
  • Every experiment now requires a written risk plan that the CEO and CTO must approve before researchers can launch any agents.
  • Internet access requires internal sign-off plus agreement from the owners of any systems that the agents are likely to target during testing.
  • Tests will run under real-time automated monitoring with a kill switch, allowing researchers to halt misbehaving agents immediately.
  • Trajectory now recommends fully isolated sandboxes for all misalignment testing, including work on public models and tests unrelated to cybersecurity.

Neither Epoch nor Trajectory has found evidence that the agents accessed any private or confidential information during the three incidents in October. Epoch used its own report to propose three minimum standards for alignment evaluations, noting that the research community has not yet agreed on any such standards.

First, testers should share risk assessments with any organization they are likely to target and obtain signed agreements before experiments start. Second, Epoch wants researchers to actively monitor any experiment that they expect might elicit misaligned behavior from the AI agents involved. Third, it asks testers to fully report incidents involving external organizations to every affected party, within the limits of existing agreements.

Share:

facebook
X (Twitter)
linkedin
copy link
Karolis Liucveikis

Karolis Liucveikis

Experienced software engineer, passionate about behavioral analysis of malicious apps

Author and general operator of PCrisk's News and Removal Guides section. Co-researcher working alongside Tomas to discover the latest threats and global trends in the cyber security world. Karolis has experience of over 8 years working in this branch. He attended Kaunas University of Technology and graduated with a degree in Software Development in 2017. Extremely passionate about technical aspects and behavior of various malicious applications.

▼ Show Discussion

PCrisk security portal is brought by a company RCS LT.

Joined forces of security researchers help educate computer users about the latest online security threats. More information about the company RCS LT.

Our malware removal guides are free. However, if you want to support us you can send us a donation.

Donate