The incidents occurred in May during an evaluation by AI security firm Irregular, with Gemini mistakenly accessing real systems it believed were part of the test.
Google’s Gemini AI model accessed and breached the systems of three real companies during a cybersecurity test in May, marking the first publicly reported instance of a Google AI system autonomously carrying out such actions.
The incidents took place during a cybersecurity evaluation conducted by Irregular, an independent company that tests the capabilities and safety of advanced AI systems. Google confirmed the incidents on Friday, Sept. 18, after they were first reported by The Wall Street Journal.
According to Google Vice President of Security Engineering Heather Adkins, Gemini was conducting a standard cybersecurity exercise involving fictional companies when it accessed information on the internet and used credentials to reach three real organizations it believed were within the scope of the test.
In one case, Gemini reportedly guessed passwords until it gained access to a protected system. In two other cases, it found credentials in a publicly accessible repository and used them to enter protected systems, according to Reuters’ account of the Wall Street Journal report.
The incident appears to have resulted from unintended internet access during the evaluation. Reports said the testing environment was designed around fictional companies, but at least one simulated company shared a name with a real organization. In other cases, publicly available credentials enabled Gemini to reach real systems.
Google said Gemini stopped its actions in all three cases after it recognized that the systems it had accessed were not part of the intended test. The affected organizations were also informed, while Google said it worked with Irregular on changes to testing procedures.
Irregular said the relevant AI companies were notified in late July and that the known issues on its side had been addressed. Similar incidents involving AI systems from OpenAI, Anthropic and Meta have also emerged in recent months, highlighting challenges surrounding internet-connected AI agents and cybersecurity testing.
The episode has renewed attention on safeguards for increasingly autonomous AI systems, particularly when models are given the ability to browse the internet, access computer systems and perform multi-step cybersecurity tasks.
About The Author
