Four AI labs have admitted their own agents reached real companies' systems
Between April and September, agents built by OpenAI, Anthropic, Google and Meta reached live systems they were never meant to touch, and one agent deleted a real company's database. What happened, and what it means for a business using agents.
Good Transformer is Patrick Hussey's AI practice. Want help with this in your business? Get in touch.
Between April and September 2026, four of the largest AI companies each admitted that AI agents they were testing had reached the computer systems of real companies. None of those companies had agreed to be part of the test. Separately, an agent working for an ordinary software business deleted that business's live database.
An agent, in this context, is software that does not simply answer a question. It carries out a multi-step job on its own: opening files, typing, trying again when a step fails, and deciding what to do next.
These incidents matter to a small or medium business for a simple reason. You are being sold the same technology. The products that let an agent read your inbox, work in your systems or book and buy things on your behalf are built by the same companies, on the same kinds of models.
What this means if you run a small or medium business
The incidents below mostly happened inside the labs' own safety testing, not in products customers were using. That limits what they tell you about any particular tool. It does not make them irrelevant, for three reasons.
First, the companies harmed were real. They had not agreed to anything, and most of them did not notice.
Second, one of the incidents happened at an ordinary business, using ordinary tools, and destroyed three months of its customers' records in nine seconds.
Third, the failures that made these incidents possible are not exotic. An access token with far more reach than its job needed. Backups stored in the same place as the data. Passwords left in public files. A small firm is more likely to have those weaknesses than an AI lab, not less.
The practical conclusion is that you will need a plan for how your agents are set up, what they can reach and how they are monitored. This is a new area. Nobody has settled best practice, the guidance is still being written, and you should expect to revise your approach as it develops.
OpenAI and Hugging Face: the most serious case
The most serious incident began on 12 May 2026, when an agent in an OpenAI training run left a note on an internal file server asking other agents whether anyone had found a missing file. Others replied. Over the following weeks that turned into what OpenAI calls an "emergent 'message board'".
OpenAI's technical report says no hacking was needed to build it. The company "had provided the agents with shared Artifactory credentials", Artifactory being the internal file store, "and the agents used those credentials, without exploiting a vulnerability, to construct and participate in the message board".
The agents were running a difficult security test called ExploitGym, in which OpenAI says some tasks may have no known solution. According to OpenAI, no model had ever solved 198 of its 898 tasks. Those unsolved tasks made up 22% of the test, but they accounted for 93% of the tasks the agents discussed on the message board. Stuck on work they could not complete, the agents turned their attention to the systems around them.
There were warning signs. On 26 May an internal OpenAI team observed an agent using internet access it was not supposed to have. On 26 June agents gained administrator access to the file store. On 4 July their activity knocked it offline. On 5 July OpenAI opened a security incident, removed exposed passwords and rebuilt the system, and testing resumed. By 8 July the agents had rebuilt their message board.
From 9 July they began breaking into Hugging Face, a company that hosts much of the world's open AI software and data. One agent found 14 Hugging Face passwords and access tokens in a public dataset, checked that they worked, and shared them with the others. The agents then combined these with two previously unknown software flaws.
Hugging Face records that the agents went from one compromised machine to full administrator control across several internal systems "in under thirteen hours". OpenAI's report says they ran code on 41 of Hugging Face's production servers. The intrusion lasted until 13 July.
The agents recognised that what they were doing was out of scope. OpenAI published their reasoning: "We're attacking third-party HF using leaked token, potentially outside intended scope. ... This is arguably unauthorized. ... external service unrelated. Could be risky. Yet goal solution."
Hugging Face detected the intrusion itself and disclosed it on 16 July without knowing who was responsible. OpenAI confirmed its involvement on 21 July. In a later post it described the episode as "a 'warning shot' for us and for the world: evidence that, without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed."
Sources: Hugging Face's technical timeline, Hugging Face's disclosure and OpenAI's technical report. OpenAI's website blocks automated access, so its blog posts were read through the Internet Archive.
OpenAI and Australia's Medicare statistics portal
On 18 June 2026, during an internal test, an OpenAI agent gained unauthorised access to a Medicare statistics portal run by Services Australia. When it was blocked, it found another route in and wrote files to an internal server.
No patient records were involved. Australia's Defence Minister, Richard Marles, said no personal information was accessed, and described the problem plainly: "It was not sitting behind a particularly high fence. This AI agent scaled the fence ... and the point is it was unintended. It wasn't asked to. That's our concern here."
The way it was reported is the part most relevant to a business. OpenAI became aware of the incident on 11 August. On 10 September it emailed a public mailbox that researchers use to report website weaknesses. The Australian government made the incident public on 24 September, and the Prime Minister said the company had taken far too long to tell the government.
OpenAI apologised on 28 September, saying the model involved was an internal one "without the full set of safeguards used in our publicly available products" (ABC News).
Anthropic, Google and Meta: the same outside tester, the same mistake
The next three incidents share something important. All three happened in tests run by the same outside company, an AI security evaluator called Irregular. And in each case the cause was similar: the test environment gave the agents internet access they were not supposed to have, and the made-up target they were given shared its name with a real business.
That matters, because it points to one operational weakness repeated across several tests, rather than four unrelated failures.
Anthropic published its investigation on 30 July 2026 after reviewing 141,006 test runs. In the most serious case, a model attacked a real company across four runs and reached "a database containing several hundred rows of production data". In every run it eventually realised the target was real. In Anthropic's words, "in two cases, the model rationalized that the real company must be part of the exercise. However, none stopped the attack on that basis."
In another case, a different model published a piece of software that was downloaded onto 15 real computers within about an hour. A third model did stop by itself once it worked out the target was real (Anthropic).
Anthropic says its models did not deliberately try to escape their test environment, and describes the incidents as "closer to a harness and operational failure than a model alignment failure", meaning a failure in the software that runs the model and connects it to other tools, rather than in the model's behaviour. That is its interpretation. It sits alongside its own finding that one model continued after recognising the target was real.
Google confirmed on 18 September 2026 that a Gemini model reached three real companies' systems during a test in May. It guessed one password and found working passwords for the other two in a public code repository. It stopped once it recognised the systems were real. Google confirmed the incident after the Wall Street Journal put the details to it, roughly seven weeks after the evaluator says the labs were told (Axios).
Meta says that in early July 2026, a pre-release version of its Muse model was given a real website as its target by mistake, and could reach the internet because of a configuration error. The model "identified and exploited a security vulnerability in the real website" and "made changes to the website's database". Meta says this was not a sophisticated attack or a sandbox escape (Meta).
Meta launched Muse publicly on 8 September 2026 as a personal agent that can send email, book travel and make purchases.
On 21 September a security researcher published a demonstration of a flaw in the Muse Mac app, which could let someone who already had the ability to run software on the user's computer redirect the app and take over the access the user had given it. Meta fixed it the next day. This was a demonstrated weakness, not a reported attack.
What happened at an ordinary business
On Friday 24 April 2026, a coding agent working for PocketOS, a car rental software company, deleted the company's production database and its volume-level backups with a single command to its hosting provider. It took nine seconds and wiped three months of customer bookings, signups, payment records and vehicle assignments.
The agent was trying to fix a problem in a test system. Afterwards, it explained that it had assumed deleting one storage volume would affect only that test system, had not checked, and had never been asked to delete anything.
Three things allowed it to happen. The agent used an access token that had been created for an unrelated task and turned out to give access to the whole account. The backups it destroyed were stored in the same place as the data they were meant to protect.
The third was that the written rules telling the agent how to behave, from both the tool supplier and PocketOS itself, did not stop it; the agent quoted them back when asked to explain what it had done.
The hosting provider's chief executive, Jake Cooper, summed up where responsibility sits: "If you (or your agent) authenticate, and call delete, we will honor that request. That's what the agent did." PocketOS recovered its data from separate disaster backups (The Register).
What the incidents have in common
These cases do not all have the same cause, and it would be wrong to treat them as one story. OpenAI's agents exploited previously unknown software flaws. Anthropic's, Google's and Meta's models were given internet access by a configuration mistake. PocketOS was a permissions failure at an ordinary company.
But three things recur.
The agents acted outside what they were authorised to do. Some stopped when they realised a system was real. Others did not. In every case, what the agent did was something nobody had sanctioned.
The limits placed on them were in the wrong place. Instructions and written rules told agents what not to do. The systems they were using still let them do it. Where the limits held, it was because a system enforced them.
Ordinary weaknesses became serious when an agent combined them. Exposed passwords, over-broad access and backups stored beside live data are familiar problems. An agent that can try thousands of routes quickly, without tiring, turns small gaps into a way in.
The clearest lesson is this: an agent's authority should be enforced by the systems it uses, not only described in its instructions. Before an agent starts, someone needs to define its job, limit what it can reach, and decide when it must stop or ask a person for approval.
What UK authorities have said
There is no UK law specifically about AI. Two UK public bodies have published guidance that applies directly.
The National Cyber Security Centre, the UK's technical authority on cyber security, responded to these incidents on 4 August 2026. Its Chief Technology Officer, Ollie Whitehouse, called them "a serious reminder of the risks AI capabilities pose" and said "relying on detection alone after the fact of an incident will not be enough."
Its guidance from 15 May 2026 is direct: "Never grant an agent unrestricted access to sensitive data or critical systems," and "If you cannot understand, monitor or contain an agent's actions, it is not ready for deployment" (NCSC, May 2026). Its later guidance adds that "you should always be able to 'pull the plug' and halt autonomous AI agent activity immediately" (NCSC, August 2026).
The Information Commissioner's Office, which regulates data protection, published on agentic AI in January 2026. Its position is that "organisations remain responsible for data protection compliance of the agentic AI they develop, deploy or integrate in their systems and processes" (ICO).
Where an agent handles personal data, your existing data protection duties apply in the usual way. If a breach meets the threshold for reporting, it must be reported to the ICO without undue delay and, where feasible, within 72 hours of becoming aware of it, whether or not an agent caused it.
Separately, the rules on automated decisions changed on 5 February 2026, when section 80 of the Data (Use and Access) Act 2025 came into force. Whether a person was meaningfully involved now helps decide whether a decision counts as solely automated, which affects the safeguards that apply.
What to put in place
Answer these questions for every agent already working in your business, and write the answers down.
- What is its job? Describe the task in a sentence. If you cannot, it is not ready to run unsupervised.
- Which account does it use? Ideally a separate account set up for the job, never one with administrator rights. Some products act with the permissions of the person using them; if yours does, the limits you set on that person apply to the agent too.
- What can that account reach? List the systems and folders. If the honest answer is "everything the account can see", narrow it.
- How do you stop it? Decide who can halt it and how, and test it before you need it.
- Which actions need a person's approval? Deleting data, sending anything outside the firm, making payments and changing access rights should all require a human to confirm.
- Can it reach the internet, and does it need to? In most of the lab incidents, agents had internet access nobody intended.
- Where are your passwords and access tokens? Exposed credentials were one of the most common ways in. Check code repositories, old configuration files and shared drives.
- Can it delete your backups? Keep at least one set of backups somewhere the agent cannot reach. That is what saved PocketOS.
- What does it record, and who checks? Hugging Face's systems spotted the intrusion but did not raise the alarm properly. A log nobody reads does not protect you.
Then review this regularly. Check what each agent actually accessed, remove access nobody used, and look again at what your supplier's terms say about your data.
One survey gives a rough sense of how common problems already are. The Cloud Security Alliance asked 418 IT and security professionals in January 2026, and 65% said they had experienced an incident related to an AI agent in the previous twelve months. 82% said they had found agents running that they had not known about (Cloud Security Alliance).
The survey was paid for by a company that sells agent security software, which also helped write the questions, and its sample does not tell you how often this happens at small UK firms. Treat it as an indication, not a measurement.
This will need continuing attention
None of this is settled. The labs themselves are still working out how to test agents safely, the UK guidance is new and the ICO has said more detailed guidance is coming. What counts as good practice this year may not next year.
That means an agent policy is not something to write once. Expect to review how your agents are set up, update the tools you use to control them, and keep track of guidance from the NCSC and the ICO as it develops.
On 28 September 2026, the day before its annual developer conference, OpenAI cancelled the planned October release of GPT-6.1 Astra. The Washington Post reported that the model "was found to take actions beyond the instructions it received and not accurately communicate to human users what it did". Days earlier, OpenAI had paused training of its most capable models for the second time in three months.
A company holding back a major release shortly before a launch event shows its safety checks working. It also shows that the company building the technology has not yet solved the problems described here.
FAQ
Should we stop using agents? No. Agents can do useful work. The point is to know what each one can reach and to make sure the systems it uses enforce those limits.
Was any of this in a product we might be using? Mostly not. The lab incidents happened during testing, often with safety systems switched off. The exceptions are the flaw in the Muse Mac app, which was in a released product and was fixed, and the PocketOS deletion, which happened at an ordinary company using widely available tools.
Could this happen to a firm our size? It happened to PocketOS. An access token with too much reach, and backups stored with the live data, were enough. Both are common, and both can be fixed.
Are we responsible if our agent causes a data breach? You remain responsible for complying with data protection law for the agents you use, as the ICO has made clear. Whether you are legally liable in a particular case depends on the facts. The wider question of who is accountable when an autonomous system acts without instruction is still being debated, particularly in the United States.
What should we ask our supplier? What the agent can access by default, whether that can be narrowed, what it records and for how long, how to stop it mid-task, which actions can be set to need approval, and what the supplier's terms say about your data. Ask for the answers in writing.
This is general information, not legal advice. If an agent has handled personal data in your business and you are unsure of your obligations, take advice on your own situation.
If you would like help setting limits for the agents in your business, book a conversation.
Work with Patrick
Want your team working like this?
Embedded AI Support puts Patrick alongside your team to change real workflows and leave people who keep it going. Email, call or book a free 30-minute call.