Can I run an LLM without a GPU?

Small quantized models, yes — and slowly. A 7–8B model quantized to 4-bit runs on CPU with enough RAM, but you'll measure output in a few tokens per second, not the snappy stream you get from an API. Fine for batch jobs and background tasks, painful for interactive chat.

How much RAM for local inference?

The model has to fit in memory plus overhead. A 7–8B 4-bit model wants ~6–8 GB; 13B around 10–16 GB; 30B-class models push 24–48 GB quantized. Add room for context and anything else on the box. This is why local inference lands in high-memory territory fast.

Ollama is the easy on-ramp — one command, model pulled, running. vLLM is for throughput and serving, more setup, better under load. For a private single-user box, Ollama is usually the right call; vLLM when you're actually serving requests at volume.

Why self-host inference at all if it's slower?

Privacy. Your prompts and outputs never touch a provider's servers or logs. For sensitive data — legal, medical, internal code, anything you can't send to a third party — a slower private model beats a fast one that sees everything. Pair it with no-KYC crypto payment and the whole chain stays yours.

Hosting local LLM inference privately: what a CPU VPS can and can't do

Let's be honest up front: if you want fast, cheap, high-quality generation, call an API. A CPU VPS will not beat a datacenter full of GPUs, and anyone telling you otherwise is selling something.

So why self-host inference at all? One reason, and it's a good one: the model on your server never sends your prompts anywhere.

What CPU inference actually looks like

You can run real models on CPU with enough RAM. A 7–8B model quantized to 4-bit works. A 13B works. You can even push a 30B-class model if you've got the memory. What you can't do is make it fast — output comes at a few tokens per second, not the instant stream an API gives you.

That's the honest trade. For interactive chat it's frustrating. For background work — summarizing documents overnight, classifying a queue, enriching data on a schedule — a few tokens per second is completely fine, and nobody's watching the clock.

The RAM math

The model has to sit in memory, quantized or not, plus overhead for context and the runtime:

7–8B, 4-bit — around 6–8 GB. Runs on a mid plan.
13B, 4-bit — roughly 10–16 GB.
30B-class, quantized — 24–48 GB, depending on quantization.
Bigger, or higher precision — you're into 64–80 GB fast — and past 80 GB no single Pro box fits, still CPU-slow.

This is why "run a local model" quietly becomes a high-memory question. The model is the memory footprint. Add a RAG index or agents on the same box and the numbers stack.

Ollama vs vLLM, briefly

Ollama is the easy door in — install, ollama run, done. It's the right tool for a private single-user setup where you just want the model available. vLLM is built for serving throughput: more setup, better under concurrent load, worth it when you're actually handling volume. Start with Ollama; reach for vLLM when you're serving real traffic.

Where private-first genuinely wins

The case for self-hosted inference isn't speed or cost — it's that some data can't leave. Legal documents. Medical records. Proprietary code. Anything where sending the prompt to a third-party API is off the table for policy or trust reasons. A slower model that runs entirely on your machine beats a fast one that logs everything you send it.

And if the data is that sensitive, the payment probably should be private too. Renting the box with crypto and no KYC keeps the whole chain — server, model, prompts, billing — off anyone's identity records. That's the actual pitch: not "cheaper inference," but "inference nobody else can see."

Bottom line

For speed and quality, use an API — no shame in it. Self-host when privacy is the requirement and slower is acceptable. Size for the model plus its context plus anything else sharing the box, and don't expect GPU speed from CPU.

When the model needs real memory, the Pro line runs 32 to 80 GB with a dedicated IP and nightly backups — enough to hold a serious quantized model with room for context around it.