Running AI Models on Your Own Hardware: When It Is Actually Worth It

The case for running a model on your own machine is privacy and cost, and both arguments are usually had at the wrong altitude. The question is not whether a local model can write a paragraph. It can. The question is whether the specific job you want to move off an API survives the drop in capability, and whether your hardware has enough memory to hold a model good enough to do it.

That second part is not a matter of opinion. It is arithmetic, and you can do it before downloading anything.

The sum that decides it

Models are distributed quantised — the weights stored at reduced precision so they take less memory. The GGUF format that nearly every local tool reads describes each of these levels precisely, and the four-bit variants most people use work out at around 4.5 bits per weight once the block scales are counted.

That gives you a rule you can apply to any model you see mentioned:

Gigabytes of memory needed is roughly the parameter count in billions multiplied by 0.55, plus headroom for context.

An eight-billion-parameter model is therefore about 4.5 GB. Fourteen billion is about 8 GB. Thirty-two billion is around 18 GB. Seventy billion is around 39 GB, which is why it stays out of reach on most laptops no matter how fast the processor is.

Machine	Comfortable model size	What that is good for
8 GB memory, no discrete GPU	Up to about 7B	Extraction, classification, short rewriting
16 GB memory or 8 GB VRAM	Up to about 14B	Summarising, drafting, routine code edits
32 GB unified memory	Up to about 32B	Most day-to-day work, with real quality
64 GB, or a 24 GB card	70B at four bits, tightly	Work where the quality ceiling matters

Apple Silicon is disproportionately good at this, because memory is shared between processor and graphics rather than fenced off on a card. A machine with 32 GB of unified memory runs models that would otherwise need an expensive discrete GPU. If you are buying hardware with this in mind, buy memory before anything else — it is the only specification that changes what you can run at all, rather than how fast you run it.

Context is not free either

The weights are the fixed cost. The context window is the variable one, and it is the part people forget. Every token you feed the model occupies space in the key-value cache, and that cache grows with the length of the conversation.

The practical effect is that a model advertised with a very large context window will not reach it on your machine. You will hit an out-of-memory error, or the tool will quietly start paging to disk and the whole thing will slow to a crawl. Set the context length deliberately to what the task actually needs rather than to the maximum, and you will have avoided the most common cause of the complaint that local models are unusably slow.

The three tools, and what separates them

Ollama is where most people should start. Pulling and running a model is a single command, it exposes an API in a familiar shape so existing code can point at it with a changed base URL, and it manages the model files for you. It has since grown a cloud offering alongside the local one, which is convenient and worth being deliberate about — for many readers the entire point is that inference happens on their own hardware, so check which mode you are in.

LM Studio is the desktop application answer: a browser for models, a chat interface, local speech transcription that does not upload audio, and a server mode when you want other software to point at it. If the terminal is a barrier for you or for colleagues you are trying to bring along, this is the one that removes it.

llama.cpp is the engine a good deal of the category is built on. Plain C and C++ with no dependencies, and acceleration for Apple Metal, NVIDIA CUDA, AMD, Intel, Vulkan and a long list besides. You reach for it directly when you want to choose the quantisation yourself, run on unusual hardware, or embed inference in something you are building. It covers everything from one-and-a-half-bit to eight-bit integer quantisation, which is precisely the dial the other two tools are setting on your behalf.

Which quantisation to pick

Four bits is the answer for almost everyone, and it is where most defaults already sit. The quality loss against full precision is real but modest, and on the work local models are actually good at you would have difficulty picking it out blind.

Below four bits the losses stop being subtle. Instructions get dropped, output formatting drifts halfway through, and the model becomes noticeably more willing to invent a detail rather than admit it does not have one. The temptation is to run a much larger model at two bits to fit the hardware. Resist it: a smaller model at four bits is almost always the better machine for the same memory.

Above four bits — six or eight — you are paying a great deal of memory for a small gain. Worth it only when you have memory spare and the task is unusually sensitive to precision.

Where local genuinely wins

Three cases. They are narrower than the enthusiasm suggests and completely real.

• Data that is not allowed to leave. Client records under a contract that names where data lives, health or legal material, anything where the review process for sending text to a third party costs more than the work itself. Local inference removes the question rather than answering it.
• High-volume repetitive work. Classifying ten thousand support tickets, pulling fields out of a pile of documents, generating first-pass alt text for an image library. These are exactly the tasks a small model handles well, and exactly where per-token pricing quietly adds up. Run them locally and the marginal cost is your electricity.
• Anything that must not break. No network, no rate limit, no model deprecated with three months' notice halfway through a project. A model file on your disk behaves the same way in two years as it does today.

Where they still lose

Hard reasoning is the honest gap. Problems with several dependent steps, where getting step two wrong quietly poisons step five, are where frontier models remain clearly ahead. Long-context work is the second — not because local models lack the advertised window, but because you cannot afford the memory to use it.

The third is more of a warning than a limitation. A local model fails differently. Rather than refusing or flagging uncertainty, it tends to produce something confident and slightly wrong. If your workflow has no review step, that failure mode will cost you more than the API ever would. This is the same judgement as deciding which tickets are safe to hand to an agent: the question is never capability in the abstract, but what happens when the output is wrong and nobody checks.

An hour that will settle it for you

Rather than reading comparisons, run this. It takes about an hour and gives you an answer specific to your own work.

1. Do the memory sum for your machine and pick the largest four-bit model that fits with room to spare. Do not start at the top.
2. Install Ollama and pull that model. Two commands. If the terminal is not for you, LM Studio reaches the same place with a download.
3. Take ten real examples of one repetitive task you already do — not a demo prompt. Ten support emails to classify, ten paragraphs to shorten, ten functions to document.
4. Run all ten through the local model and through whatever you use now. Score them yourself, quickly, out of five.
5. Compare the totals. If the local model lands within a point or so, that task can move, and every future run of it is free.

The result is nearly always the same shape: some of your work moves and some does not, and the split runs along the boundary between mechanical tasks and judgement. That is a better outcome than either camp advertises. If you are still deciding which hosted model to keep for the difficult half, our comparison of the main assistants for coding work covers that side of the split.

What we would not do is treat this as a replacement decision. The people getting the most out of local models are running them alongside a hosted one, not instead of it, and the skill worth building is knowing which pile a given job belongs in.