Self-hosting open LLMs: Llama and friends

For the first year of the LLM boom, building an AI feature meant calling someone else’s API — OpenAI’s, mostly. Your data went to their servers, you paid per token, and the model was a black box you rented. That was the only option for a model worth using. It isn’t anymore. Over the past several months a wave of open-weight models — Llama 2, Mistral, Mixtral and a growing crowd of others — have become good enough that running a capable LLM on your own infrastructure is a real choice. The weights are published; you can download them and run them on your own hardware. This post is about what that opens up, what it actually costs you, and the honest question hiding underneath the excitement: just because you can self-host, should you?

What “open weights” actually gives you

First, a precise framing, because “open” is doing a lot of work and not all of it is what people assume. These models ship their weights — the trained parameters — so you can run the model yourself. That’s a genuine and important freedom, and it’s distinct from the model being “open source” in the full sense: the licences vary, the training data generally isn’t published, and “open weights” mainly means you can run it, not that you can fully reproduce or audit it. Still, being able to download a capable model and run it on your own machine, with no API call leaving your network, is a real shift from the rented-black-box world of a year ago. The question is what that shift is for.

Why you’d want to self-host

There are three real motivations, and it’s worth being clear-eyed about each:

Privacy and data residency. This is the big one. When you call a hosted API, your prompts — which may contain customer data, proprietary information, regulated content — leave your infrastructure and go to a third party. For some applications that’s fine; for others it’s a dealbreaker, whether because of regulation (healthcare, legal, certain EU data-residency requirements), contractual obligations, or simply not wanting sensitive data to leave your walls. A self-hosted model keeps every prompt and response inside your own infrastructure. For a lot of teams, this alone is the whole reason — it’s not about cost or quality, it’s that the data cannot leave.
Cost at scale. Per-token API pricing is cheap to start and can get expensive at volume. If you’re processing millions of requests, the API bill can dwarf the cost of running your own hardware. Self- hosting trades a per-token operating cost for a fixed infrastructure cost, which wins past a certain volume — but mind the word “past,” because below that volume it loses, often badly.
Control and stability. A hosted model can change under you — get updated, deprecated, retired — and your carefully-tuned prompts may behave differently overnight. A model whose weights you’ve downloaded is frozen: it is exactly the same model tomorrow as today, forever, because it’s a file you hold. For features that need stable, reproducible behaviour, that permanence is valuable.

These are real benefits, and for the right application any one of them justifies self-hosting. Privacy especially is often non-negotiable in a way that ends the discussion before cost or quality enter it.

What it actually costs you

Here’s the part the enthusiasm tends to skip, and it’s the part that should drive the decision: you become the operator of an LLM, and that is a real, ongoing commitment, not a one-time setup. The honest ledger:

You need the hardware. Capable models want a serious GPU (or several), and good GPUs are expensive to buy and non-trivial to rent. The “free” model isn’t free; you’ve moved the cost from a per-token line item to a hardware-and-power line item, and for low volume that trade is a loss.
You run the infrastructure. Serving a model means standing up and maintaining inference servers, handling scaling, monitoring, updates, failures — all the operational work the hosted API was doing for you, invisibly, in exchange for the per-token price. That work doesn’t vanish when you self-host; it moves onto your plate.
There’s a quality gap to be honest about. As of early 2024, the best open models (Mixtral and the strong Llama 2 variants) are genuinely good and have closed a lot of distance — but the very top hosted models still lead on the hardest tasks. For many applications the open models are more than good enough; for some, the gap matters. You have to evaluate your task on the candidate model, not trust a leaderboard, because “good enough” is entirely task-dependent.

None of this is a reason not to self-host. It’s the reason to self-host deliberately, with the ops cost counted honestly up front rather than discovered later.

The tooling has gotten genuinely good

The one thing that’s clearly improved is how approachable running these models has become. A year ago this was research-grade fiddling; now tools like Ollama and llama.cpp let you run an open model locally with very little ceremony, and quantization — compressing a model’s weights to use less memory at a small quality cost — means models that once needed a data-centre GPU can run on much more modest hardware, even a well-specced laptop for the smaller ones. This is what’s turned self-hosting from a research project into a practical option: you can try a model on your own machine in an evening and get a real feel for whether it’s good enough for your task before committing any infrastructure. That low-cost evaluation path is genuinely valuable, whatever you decide.

So should you?

Here’s the honest answer, and it’s the same shape as every “should I adopt this” question in this series: it depends on what you actually need, and the default for most teams is still the hosted API. Self-hosting is the right call when you have a specific driver for it — most often a hard privacy or data-residency requirement that takes the hosted API off the table entirely, sometimes genuine cost pressure at high volume, sometimes a need for frozen, reproducible behaviour. When one of those applies, self-hosting is not just defensible but correct, and the tooling now makes it practical.

But self-hosting because you can, or because owning the model feels better than renting it, is the over-engineering trap wearing new clothes. If you don’t have a privacy requirement that forbids the API, aren’t at the volume where the economics flip, and don’t need frozen behaviour, then the hosted API is almost certainly the better engineering choice — you get a stronger model and you don’t sign up to operate GPU infrastructure for a problem you didn’t have. The right move is to match the tool to the need: identify whether you have a real driver for self-hosting, and if you don’t, happily keep calling the API and spend your operational budget on something that actually moves your product.

Verdict

Open-weight models — Llama 2, Mistral, Mixtral and the growing field around them — have made self-hosting a capable LLM a genuine option for the first time, and the tooling (Ollama, llama.cpp, quantization) has made it practical enough to evaluate in an evening. The real reasons to do it are specific and strong: privacy and data residency (often the dealbreaker that ends the discussion), cost at high volume, and frozen, reproducible behaviour you fully control. But the honest ledger is that self-hosting makes you the operator of an LLM — you buy the GPUs, run the inference infrastructure, and accept a quality gap against the top hosted models that may or may not matter for your task. So the decision is the familiar one: self-host when you have a concrete driver — most often a hard privacy requirement that takes the API off the table — and otherwise stick with the hosted API, which for most teams at most volumes is still the better engineering choice. Don’t self-host because you can; self-host because you must, or because the economics genuinely favour it. Match the tool to the need, and you’ll know which side of that line you’re on.