Local-first AI: running models at the edge

For the whole first era of the LLM boom, “using AI” meant calling a big model in someone’s data centre over the network. That assumption is breaking down. Two things changed at once: small models got genuinely good — capable open models that fit in a few gigabytes now do real work — and devices got powerful enough to run them, with phones, laptops, and even browsers gaining the hardware and runtimes to execute a model locally. The result is a real movement toward local-first AI: running models at the edge, on the user’s own device, instead of always reaching back to a server. This is a trend post on what that shift buys you, what it costs, and why the honest answer for most applications isn’t “local” or “cloud” but a deliberate mix of both.

What “at the edge” means here

Let me pin down the term, because “edge” gets used loosely. Here it means running the model on or near the user’s own device rather than in a central cloud: directly on a phone or laptop, in the browser via the runtimes that now support it, or on local hardware the user controls. The model executes where the user is, and the inference happens without a round-trip to a provider’s servers. That physical relocation of where the computation happens — from a distant data centre to the user’s own silicon — is the whole basis for the benefits and the costs that follow.

Why local-first is appealing

The case for running models locally is genuinely strong on several axes, and each is a real advantage the cloud model can’t match:

• Privacy. This is the headline. When the model runs on the user’s device, their data — the prompt, the document, whatever sensitive thing they’re working with — never leaves the device. No data sent to a third party, no prompt sitting in someone’s server logs. For privacy-sensitive applications this is transformative, and it’s the same driver behind self-hosting, now taken all the way to the user’s own hardware. It’s the strongest form of “the data cannot leave.”
• Latency. No network round-trip means no network latency. For a small model on capable hardware, a local inference can return faster than even a fast network call to a remote model would — and it works with no perceptible lag for interactive features. Local is often the fast option, not just the private one.
• Offline capability. A local model works with no internet connection at all. The feature keeps functioning on a plane, in a dead zone, anywhere connectivity is poor or absent — which is impossible for a cloud-dependent feature. For some applications this alone is the reason.
• Cost. Inference on the user’s device costs you nothing per call — there’s no per-token bill, because you’re using the user’s compute, not paying a provider. At scale, moving inference to the edge can dramatically change the economics of an AI feature.

Privacy, latency, offline, cost — that’s a compelling list, and for the right application any one of them justifies going local. The catch is what you give up to get them.

The honest trade-offs

Local-first isn’t free, and the costs are exactly the mirror of the benefits — this is the match-the-tool- to-the-need decision in a new domain:

• Capability gap. The models that fit and run on a device are smaller and less capable than the frontier models in the cloud. They’ve improved enormously, and for many tasks they’re more than good enough — but for the hardest reasoning and generation, the big cloud models still lead by a clear margin. You trade capability for locality, and whether that trade is acceptable is entirely task-dependent.
• Device constraints and variability. Running a model uses the device’s memory, compute, and battery, and — critically — your users’ devices vary enormously. A model that runs well on a high-end laptop may be sluggish or impossible on an older phone. You’re targeting a wide range of hardware you don’t control, which a uniform cloud backend never forces you to think about.
• Distribution and updates. The model has to get onto the device (a download, often a large one) and be updated there, rather than you updating one central deployment. Shipping and maintaining a model across many devices is real logistics that the cloud model hands you for free.

None of these kill local-first; they define where it fits. The capability gap means it suits tasks a small model handles well; the device variability means you must design for a range of hardware; the distribution cost means the model has to be worth shipping.

The hybrid reality

Here’s the honest conclusion, and it’s the same shape as every “which approach” answer in this series: for most applications the right answer isn’t local or cloud — it’s a deliberate mix. The patterns that actually win:

• Local for the common, simple, privacy-sensitive case; cloud for the hard case. Run a small local model for the frequent, easy, latency- or privacy-critical interactions, and fall back to a powerful cloud model for the harder tasks the local model can’t handle well. The user gets fast, private handling of most requests and full capability when it’s genuinely needed.
• Local for privacy-critical data, cloud for the rest. Keep the sensitive processing on-device where the data can’t leave, and use the cloud where privacy isn’t the binding constraint.
• Local-first with cloud augmentation. Default to the on-device model and reach out to the cloud only when the task demands more than the local model offers — getting the privacy, latency, and offline benefits by default, with cloud capability as the escape hatch.

The deeper point is that local-first AI has expanded the design space rather than replacing the cloud. A year or two ago, “where does the model run?” had one answer; now it’s a real architectural choice with local, cloud, and hybrid on the table, and the right pick depends — as always — on the specific requirements of privacy, latency, capability, offline support, and cost for your feature.

Verdict

Small models got good enough and devices got powerful enough that local-first AI — running models at the edge, on the user’s own phone, laptop, or browser — is now a real architectural option rather than a research curiosity. Its advantages are strong and concrete: privacy (the data never leaves the device, the ultimate form of “it cannot leave”), latency (no network round-trip, often genuinely faster), offline capability (works with no connection at all), and cost (you use the user’s compute, with no per-call bill). The trade-offs are the exact mirror: a capability gap versus frontier cloud models, device constraints and huge hardware variability you must design around, and the distribution and update logistics of shipping a model to many devices. So the answer for most applications isn’t local or cloud but a deliberate hybrid — local for the common, simple, privacy-sensitive, or offline case, cloud for the hard tasks a small model can’t handle — defaulting to on-device for its benefits with cloud capability as the escape hatch. Local-first hasn’t replaced the cloud; it’s expanded the design space, and “where does the model run?” is now a real decision to make per feature, weighing privacy, latency, capability, offline, and cost. Match the placement to the need, and very often the best system uses some of both.