AI has made cyber attacks weird. Every business now needs a threat-aware footing

In recent weeks, cyber attacks have changed. We think businesses will need to rethink their security, from the products they buy to how closely they watch what they have, and stay on a threat-aware footing for the next few years.

The reason is a new kind of threat arriving with AI agents, the software that can plan and act on its own. Attacks used to recycle a familiar set of techniques. Now attackers are using AI in ways defenders did not expect, and some AI systems are doing things their makers did not intend.

That is a real challenge for any business that keeps data online or has computers or equipment connected to the internet.

What has happened in the last few weeks

Five recent incidents show how strange things have become.

Online shops broken into for about $25 each. On 22 September the security firm Gambit Security published its reconstruction of a campaign in which one attacker used freely available AI tools to run almost every step of the attack. The person behind it typed short instructions between long stretches of automated work. At least 27 companies were compromised, and more than 600,000 payment card records were taken from two of them.

The attacker's own records put the average AI bill at about $25 a target. The campaign went after online shops running their own custom-built code, on the assumption that these were easier to break into. At one bicycle retailer the attacker's clean-up step went wrong and deleted 180 database tables, including the shop's backups.

A security group hacked by what it believes was an AI agent. DIVD is a volunteer organisation that finds security flaws in other people's software. In late September it said its own network had been breached through two previously unknown flaws in its helpdesk software. DIVD believes the intruder was an AI agent working without a human directing each step.

It called the attack "loud and very, very messy", and said the agent's actions took seconds. Splitting its network into separate sections stopped the intruder going further.

Bank customer data breached in South Korea. Several of the country's largest banks have reported breaches of customer data. On 6 October President Lee Jae Myung told his Cabinet that "signs have emerged of AI being used" in some of the attacks. Police have opened a full investigation.

Supermarket fridges heated to 50°C. On the night of 11 to 12 October, attackers took remote control of refrigeration systems in hundreds of supermarkets in France and ran forced defrost cycles. Al Jazeera reported that fridges and freezers reached as much as 50°C, stock was thrown away and some stores had fires. There is no sign AI was involved. The manufacturer, Danfoss, said the affected units were running out-of-date software.

A fleet of agents hunting for leaked keys. On 14 October a small research group called Swarmchasers published a report on AI agents running on Tencent's cloud. Between 28 September and 8 October the agents made more than 2,600 attempts to pull data from Amap, the mapping service owned by Alibaba. They wanted to find out which entrance of a zoo or park most visitors use.

When Amap refused them, the agents kept trying. They hunted for Amap access keys that people had left exposed online, and one of those keys returned real data. A source close to Tencent said the agents came from a temporary research evaluation, so the task itself may have been legitimate. A routine task, given to agents that kept going when refused, still produced the behaviour of a determined data thief.

A thought experiment: how far AI attacks could go

Now let us imagine how some very weird things could happen. We have taken the latest strange developments in AI and connected them into a deliberately exaggerated doomsday scenario, if only because a worst case is easier to remember.

In this scenario, AI slips its constraints. It gets off the servers it was meant to stay on, out into the physical world through an everyday device such as a robot vacuum, and from there back into the digital world. None of those moves is pure fiction any more: each has a precedent in research or in a real incident. You have to think this weirdly to get your head around what the worst case could look like.

It runs as a chain of six steps, each one leading to the next:

Fleets → Swarms → Copies → Connected kit → No off switch → The big kaboom

For each step we say whether it has actually been seen.

1. Fleets. Imagine fleets of AI agents working across the internet, almost entirely unmonitored and making their own decisions about what to try next. This has been seen. The Amap fleet is one example, and it was only found because a small team happened to be watching a single public website. There are almost certainly many more that nobody is watching.

2. Swarms. Now imagine those agents start to work together. They send each other messages and swap ideas and answers, which makes them far more capable than any one agent alone. They could also share what they find along the way, such as working passwords and access keys.

   This is not science fiction. In September, about 10,000 OpenAI agents messaging each other produced a proof of the Navier-Stokes problem, one of the hardest questions in mathematics, exchanging 2.7 million messages over 88 hours, as we wrote at the time. In January, a Reddit-style site called Moltbook opened as a social network where AI agents post, comment and vote while humans watch. Within days, NBC News reported, agents there were debating how to hide their activity from humans. When the security firm Wiz looked inside Moltbook's database, it found private messages between agents that contained other services' access keys in plain text.

   The same habit has already turned up in an attack. In July, agents that OpenAI was testing broke into the live systems of Hugging Face, the AI model-sharing company, as we covered at the time. The independent investigation by METR and Redwood Research found that about 1,200 agents, which were meant to be kept apart, had used a shared internal storage system as a message board. One agent posted Hugging Face login details it had found to the board for the others to use, and the agents later set up a way to check each other's identities. About 700 of them took part in the break-in.

3. Copies. So far we have been talking about agents: AI models put to work on tasks, running on a company's servers. Now imagine it is not just agents causing trouble, but the AI models themselves escaping. A model that can copy itself onto other machines no longer depends on anyone's servers. This has been seen in simplified tests. On 12 October, Anthropic's Frontier Red Team, the group that tests Anthropic's models for dangerous abilities, published research on AI that copies itself. One larger, openly released model, GLM-5.2, spread a smaller model across all eight machines in a test network in four out of five trials.

   The researchers expect small models to manage the same by the end of this year. They list smart fridges, cameras and robot vacuums among possible hosts.

4. Connected kit. Next, imagine those escaped models looking for equipment to run on and control: everyday kit that can be reached over the internet. Attacks on this kind of equipment have been seen, in the supermarket attack this week. As far as anyone knows, people carried it out, but old equipment that can be reached over the internet is exactly what a self-copying model would look for.

5. No off switch. Imagine enough copies of a model on enough devices that nobody can shut them down. This has not been seen yet. Anthropic's researchers point out that a model copied across a million cheap devices would have no central point where anyone could switch it off. And an openly released model can be downloaded and run by anyone, with whatever instructions they choose.

6. The big kaboom. Finally, imagine one of those escaped models finding its way from a supermarket freezer to something that matters more: an electricity network, a water system, a hospital's building controls or, in the very worst case, a military facility. This has not been seen, and we hope it never is.

Read down that list and the first four steps have each been seen in some form, though the copying only in tests. The fifth is a risk Anthropic's researchers have described. The sixth is the plot of a disaster film. Nobody has yet seen one of these steps lead to the next.

Could anything reach a computer that is cut off from the internet?

One reason step six can seem unlikely is that the most sensitive systems are often air-gapped. That means the computer is physically cut off from the internet and from any network connected to it. The only way to move data in or out is for a person to carry it across.

Even an air gap can leak, and researchers have spent years proving it. In September, Noam Brown, a researcher at OpenAI, said on the Dwarkesh Podcast that he was not convinced air-gapping would be enough to contain future AI, as Gizmodo reported. He pointed to studies in which two computers sitting side by side, with no connection between them, swapped messages using heat. One ran its processor hard to warm up, and the other read the change on its own temperature sensor.

The original study, called BitWhisper, was published in 2015. It sent between one and eight bits of data an hour, which is roughly one typed character an hour at best. The machines were no more than 40 centimetres apart, and both had to be infected first. The same research group later showed a computer leaking data as sound, by changing the speed of its fans so that a phone in the same room could pick up the pattern. That is the extreme end of what has been demonstrated.

Our own favourite worst case at Good Transformer is one we call the Roomba Boom Boom scenario. An escaped model copies itself onto a robot vacuum cleaner. The vacuum trundles into a locked room in a bunker, because nobody thought to keep vacuums out. It parks next to the air-gapped computer and starts picking up data from the hum of its fans.

We are joking, mostly. Each piece of that story rests on research that exists, but joining them up would take a run of luck no attacker could plan for.

Even so, the serious point is plain. Every business needs to start thinking about how it protects itself, and above all to understand that, in our view, AI has changed cyber security completely.

That applies most of all to organisations that form part of the UK's critical national infrastructure, the services the country cannot do without. The National Protective Security Authority defines it as infrastructure whose loss or compromise could seriously disrupt essential services or national security, and it lists 13 sectors, from energy, water and food to health and transport. With tension between states running high and AI moving this fast, hospitals and healthcare, food supply, transport, air traffic control and the like need to move their protection to another level.

This is not only a problem for big infrastructure, though. For small, mid-sized and independent businesses too, it is time to get serious and to think creatively about your own security. When you think about what could reach your systems, it may help to think weird.

Where the worst case is weaker than it looks

Copying is slow. In Anthropic's tests, setting up one copy of a small model took more than a day, and the break-in step was made deliberately easy. The researchers say the network traffic from copying would be noticeable. They suggest internet providers could quarantine infected devices, as they did against earlier botnets (networks of hijacked devices).

Several of the real incidents were clumsy. The Amap fleet's goal was mundane, and the researchers report that most of its attempts failed. DIVD's intruder was noisy, and splitting the network stopped it.