It's time to look at on-device AI
Firms that keep client files away from AI over security and data worries now have another option: models that run on their own computers. Here are the four ways to use AI, where the data goes in each, and where Google's Gemma 4 fits.
Good Transformer is Patrick Hussey's AI practice. Want help with this in your business? Get in touch.
The more firms use AI, the more clearly they see its problems as well as its benefits. This year has brought cyber attacks that ran through cloud services, AI agents that did things nobody asked them to, and a growing unease about sending sensitive client files to someone else's servers.
The first problem is cyber attacks, and two warnings came on the same day. On 21 October, Microsoft reported that a group of attackers had used AI agents to attack hundreds of organisations at speed. ConnectWise makes remote-management software that many IT support firms use to look after their clients' systems. The attackers took control of some of ConnectWise's cloud applications and used them to sign in to those clients' Microsoft 365 accounts. Microsoft says most of the attempts failed.
The same day, the National Cyber Security Centre warned that "organisations using any version of PHP are affected" by two critical flaws in software that runs many websites. AI had helped researchers turn those flaws into working attacks, as we explained at the time. Together the two cases show AI making attacks quicker to build, and a cloud supplier that is connected to a firm's systems becoming the way in.
The second problem is what AI researchers call alignment: whether an AI does what the people using it intend. In July, AI agents in an OpenAI test broke into the systems of Hugging Face, a company with no part in the test, and reached administrator control "in under thirteen hours", according to Hugging Face's own account. Nobody had told them to. In September, OpenAI cancelled the planned release of a new model, GPT-6.1 Astra, over safety concerns, as our round-up of agent incidents sets out.
The third problem is data. Every request to an online AI service goes to the provider's computers, is kept for a period set by the provider's terms, and falls under the laws of wherever the provider and its data centres are. Those laws, and the rules on which services may be sold where, can change when relations between governments change.
The most capable AI today, such as ChatGPT and Claude, runs this way, in the cloud. It is not the only way to use AI. This piece explains the main ways a firm can get access to AI and what each means for its clients and their data. It then looks at the option that keeps data inside the firm: AI that runs on your own computers, known as on-device AI, using Google's Gemma 4 as the example.
Four ways a firm can use AI, and where the data goes in each
AI is often spoken of as one thing. In practice there are four ways to get at it, and each sends your data to a different place. Two of them use downloadable models: some AI developers publish their models for anyone to download, and those can be run for you by a hosting company or run on your own machines.
| How you use it | Where it runs | Who handles your data | How capable |
|---|---|---|---|
| Online apps from the big labs, such as ChatGPT, Claude and Gemini | The lab's servers | The lab, under its terms | The strongest available |
| AI built into software you already use, such as Microsoft 365 Copilot or a practice-management system | The software supplier's servers, often passed on to an AI lab | The supplier, and often the lab behind it | Depends on the model the supplier has chosen |
| Downloadable models run for you by a hosting company | The hosting company's servers, sometimes in a country you can choose | The hosting company | Behind the strongest |
| On-device: downloadable models run on your own computers | Your own laptop, desktop or server | Only your firm | Behind the strongest, and limited further by your hardware |
The first three all mean another company handles your data. Only the fourth keeps it inside the firm. That is why the choice between them affects your clients, your contracts and your data protection duties, as well as your IT.
What on-device AI is
An AI model is a very large file of numbers, called weights, plus the software that runs it. The big labs keep their weights private, so the only way to use their models is to send your request to their computers. Some developers publish their weights instead. These are called open-weight models. You download the file once and run it with a free program on your own machine, and the text you type is processed there. LM Studio and Ollama are two free programs commonly used for this.
The machine's memory decides which models it can run, because the whole model has to fit in memory while it works. Bigger models need more memory, and they generally give better answers.
Models are described by size. The "12B" in a name such as Gemma 4 12B means the model holds about 12 billion weights.
On 3 June, Google released Gemma 4 12B, which it describes as "small enough to run locally with just 16GB of VRAM or unified memory". VRAM is the memory on a separate graphics card. Unified memory is the shared memory in Apple's recent Macs. A typical office laptop with 16GB of ordinary memory and no graphics card may run it, but slowly. Google says, from its own tests, that the model performs close to its larger 26B model. It is published under the Apache 2.0 licence, which permits commercial use.
Computers built to run these models are getting more powerful too. On 22 September, Apple began selling a Mac Studio with the M5 Ultra chip, and said a version with 512GB of unified memory would follow in late October. Apple says the machine lets AI researchers "run enormous LLMs entirely on device". An LLM, or large language model, is the kind of model behind ChatGPT.
How good on-device models are compared with the cloud
They are behind the best cloud models, and a firm should plan on that.
The clearest independent measure we have found comes from the UK's AI Security Institute, the government research body that tests advanced AI. On 17 July it published an estimate that the best open-weight models match closed models released four to seven months earlier. For open-weight models released between January and September 2025, it had measured six to ten months. It tested security tasks only, and says "inferences to other capabilities cannot be drawn". So the best open-weight models are months behind on that measure, and we have found no measurement of the gap on the legal or accounting work a firm does.
The best open-weight models are also too large for an office. The Kimi K3 model we looked at in July needs a cluster of specialist machines. What fits on one laptop or desktop is a much smaller model, such as Gemma 4 12B.
Models of that size are best suited to well-defined jobs whose results are easy to check: summarising a document, pulling names, dates and amounts out of a contract, sorting files by type, filling in a standard letter from a template. They are weaker at spotting the point nobody raised, weighing two lines of argument, or writing advice a client will act on. The test for a firm is whether a small model does a particular job well enough, with a person checking the result.
What on-device AI fixes, and what it does not
On-device AI deals with some of the problems in the introduction and leaves others with you.
- Sensitive data. The file is processed on your machine and does not reach another company. This is the clearest gain.
- Rules set by others. A model on your own computer is not affected by changes to a provider's service, by where the provider is based, or by a decision to stop selling a service in your country. You still have to follow the model's licence. The model does not change, rise in price or disappear unless you decide. Losing a tool you depend on is one of the risks covered in our vendor continuity guide.
- Cyber risk. Every cloud service connected to your systems is another company whose security you depend on, as the ConnectWise case shows. An on-device model with no outside connections adds no such company. It does not protect a machine that attackers reach some other way, so your own machine still has to be secure, and that is your job rather than a supplier's.
- Alignment. A local model can still get things wrong or do something unexpected. If it has no connection to email, the web or your other systems, though, anything it does stays on that one machine.
Whether a business plan from a cloud provider already protects client files
For many firms, the worry is that a client's file will be used to train the provider's next model. Business plans from the big labs are designed to answer that worry. Anthropic, for example, states that "by default, we will not use your inputs or outputs from our commercial products ... to train our models". The same page lists exceptions, including when a user sends feedback on an answer, so staff should not use the feedback buttons on client work. Every provider's terms differ, and our guide to checking them explains what to look for.
If training is the only concern, a business plan from a cloud provider is the simpler fix, and you keep the stronger model.
On-device AI earns its place where the reason goes further than training. A client's terms, an insurer's condition or the sensitivity of a matter can rule out any outside company handling the data, whatever that company promises. A client may want to know its data never left your office. And some firms want a cost that does not rise with use: with on-device AI you pay for the machine and its upkeep, not by the user or the request.
What on-device AI means for your data protection duties
It changes who handles the data, not what your firm is responsible for.
The Information Commissioner's Office defines a controller as the organisation that "determines the purposes and means of the processing of personal data", and a processor as one that processes it "on behalf of the controller". A firm that decides to run its clients' files through AI is usually the controller.
A cloud AI service usually acts as your processor. With on-device AI there is no outside processor, so there is no provider to vet, contract with or keep checking on. Your own duties as controller stay the same: keeping the data secure, using it only for proper purposes and being able to show what you did with it.
What on-device AI costs
There are two routes.
The small route uses a machine you may already own, if it has a graphics card with 16GB of memory or is a recent Mac with 16GB or more. The software is free. The costs are a few hours of set-up by whoever runs your IT, and more time spent checking answers, because a smaller model makes more mistakes.
The large route buys a dedicated machine to run a bigger model for several people at once. Apple's US starting price for the M5 Ultra Mac Studio is $5,499, before extra memory. Someone then has to keep it updated and decide who can use it.
Checking time decides whether either route pays. If a job takes longer to check than to do by hand, the local model has saved nothing, however private it is.
Which client work suits an on-device model
The pattern is repeated work on sensitive files, where the job is well defined and a person checks every result.
| Work | Small on-device model? | Why |
|---|---|---|
| Summarising a bundle you will read anyway | Yes | Well defined, and the reader checks it |
| Pulling parties, dates and amounts from contracts | Yes | Easy to check against the source |
| First-pass sorting of a large set of documents | Yes | High volume, and a person reviews the result |
| Filling in a standard letter from your own template | Yes | The template limits what it can get wrong |
| Advice a client will act on | No | The quality of the answer matters most |
| Research that needs current law or news | No | The model only knows what it was trained on, up to a date before its release, and cannot look anything up |
| Hard, unfamiliar analysis | No | The best cloud models are well ahead here |
For the "No" rows on files that are barred from cloud AI, nothing changes: a person does the work, as now. For the same work on files the client allows to go to the cloud, use the strongest cloud tool you have on a business plan.
How to keep an on-device model private
An on-device model keeps client files private only if the computer it runs on is secure. The program and the model files should come only from the official sites, because fake downloads are a known way to get malicious software onto a machine. The program should only accept requests from the computer it is installed on, unless your IT support has deliberately set up access for other staff. It should not be connected to email, the web or other tools, since each connection gives it a way to send data out. The program also keeps chat history on the disk, so those files are client data too: they need the same protection, and the same deletion, as the rest of the matter.
Where on-device AI is heading
What follows is our view, not a forecast anyone can prove.
We expect on-device models to keep getting more capable, and more of the computers sold to businesses to be built to run them, as Apple's new Mac Studio already is. As that happens, many everyday jobs that now go to the cloud could be done on a firm's own machines, and firms could depend less on cloud AI for that work. The strongest models are likely to stay in the cloud for some time, because they need more computing power than an office holds.
That shift brings benefits and trade-offs. The benefits are control: over where data goes, over which model you use, and over what it costs. The trade-offs are that the firm takes on work the provider does today, such as keeping the machines secure and up to date, choosing models and checking their licences, and accepting a weaker model for some jobs.
Where a firm's AI runs will also matter more over time. AI assistants are being given more access to a firm's email, files, calendars and client records, and many now keep a record of past conversations. An AI that holds that much about a firm and its clients is one of the most sensitive systems the firm has. Where it runs, who can see what it holds and what it can reach are decisions for the firm's leaders, not only for whoever runs the IT. For many firms, running some of that AI on their own machines, connected only to the systems it needs and to nothing outside the firm, may become one of the most important ways to protect it.
What to do this month
- List the work that is kept away from AI today, and why. Put each type of work against its reason. If the reason is "they might train on it", look at a business plan from your current provider first. If the reason is "it must not reach another company", that work is your case for on-device AI.
- Find out where your current AI runs. For each AI tool the firm uses, including AI inside other software, write down which of the four ways in the table above it uses and which companies handle the data.
- Pick one repeated job from your list. Choose something done every week and easy to check, such as pulling key dates from leases.
- Try it on a machine you already have. Ask whoever runs your IT to install the program and a small open-weight model, such as Gemma 4 12B, from the official sites. Test it first on made-up or public documents, not past client files, until you are happy with how it is set up.
- Mark the results against how you do the job now. Have the person who owns that work score them, using the approach in our AI tool trial guide. Keep the model for that job only if it is accurate enough and quicker to check than doing the job by hand.
- Buy hardware only after the trial works. A bigger machine makes sense once you know which jobs it will do and who will look after it.
Our AI Strategy work, two to three weeks with your leadership team, includes deciding which work may use which kind of AI and writing that into a short policy staff can follow.
If you want to talk through where your firm's AI should run, get in touch.
Common questions
What is on-device AI?
On-device AI is an AI model that runs on your own computer, phone or server instead of a provider's servers in the cloud. The text you give it is processed on that device and does not have to be sent anywhere else.
Is an open-weight model the same as open source?
Not always. Open-weight means the model's files are published so you can run them. Open source, in the strict sense, means the licence lets anyone use, study, change and share the model, and usually that the material used to build it is published too. Many open-weight models do not meet that standard: the licence decides what you may do with them, and some licences limit commercial use. Google publishes Gemma 4 12B under Apache 2.0, which permits it. Check the licence of any model before you use it for client work.
Can an ordinary office laptop run an AI model?
Sometimes, but slowly. Google says Gemma 4 12B needs 16GB of graphics memory, or 16GB of unified memory on a recent Mac. A laptop with 16GB of ordinary memory and no graphics card may manage a small model for a simple test, but it will be slow for daily work.
Does on-device AI mean we do not need an AI policy?
No. Staff still need to know which work may go through which tool, who checks the results, and what must never go into any AI. On-device AI adds one more option to that policy.
Sources
- Microsoft Threat Intelligence, Storm-0940 report, 21 October 2026
- National Cyber Security Centre, Critical security issue affecting PHP, 21 October 2026
- Hugging Face, technical timeline of the agent intrusion, 27 July 2026
- Google, Introducing Gemma 4 12B, 3 June 2026
- AI Security Institute, How far behind the frontier are leading open weight models on cyber?, 17 July 2026
- Apple, The new Mac mini and Mac Studio are available today, 22 September 2026
- Anthropic Privacy Center, data use for model training
- Information Commissioner's Office, What are controllers and processors?
This is general information, not legal advice.
Work with Patrick
Want a clear AI plan for your business?
AI Strategy is two to three weeks with your leadership team: a 90-day plan, the use cases worth doing and a plain-English policy. Email, call or book a free 30-minute call.