Why does RAG need so much RAM?

Fast vector search wants the index resident in memory. Every document chunk becomes an embedding — a vector of a few hundred to a couple thousand floats — and at millions of chunks that adds up. Push the index to disk and search latency jumps; keeping the whole reason you self-hosted (speed + control) means keeping it in RAM.

How much RAM for a given corpus?

Rough feel: a few hundred thousand embeddings sit fine in 2–4 GB. Low millions, with the app and OS around them, and you're at 16–32 GB. Tens of millions or high-dimension vectors and you're into 48–80 GB, and past that you split across servers. Dimension and index type swing this a lot, so measure your own.

pgvector or a dedicated engine like Qdrant?

If you already run Postgres, pgvector is the least-effort path — one extension, one database. For millions of vectors with heavy filtering, a purpose-built engine earns its separate service. Start with what you already operate; move only when search gets slow.

Why self-host instead of a managed vector service?

Two real reasons: your embeddings often encode private data (docs, notes, customer content), and self-hosting keeps that on a server you control. And it's flat cost — managed services meter by vectors and queries, a VPS is one monthly number with no per-query bill.

Self-hosted RAG at scale: how much RAM a vector index really eats

Every RAG tutorial runs on a laptop with a few hundred documents, and it feels effortless. Then you point it at a real corpus — a company's docs, years of tickets, a knowledge base — and suddenly memory is the whole conversation.

RAG doesn't scale by CPU. It scales by RAM.

Why the index wants memory

Retrieval works by turning every chunk of text into an embedding — a vector, a few hundred to a couple thousand numbers long. Search means comparing your query vector against all of them, fast. "Fast" is the operative word: for low latency the index needs to live in RAM. On disk it works, but every query pays a penalty, and low-latency retrieval was the point of self-hosting in the first place.

So the memory bill scales with two things: how many chunks you have, and how wide each vector is.

Real numbers, roughly

Measure your own — dimension and index type move this a lot — but as a starting feel:

A few hundred thousand embeddings — comfortable in 2–4 GB. A personal knowledge base, a single product's docs.
Low millions — with the app, the model client, and the OS around it, plan for 16–32 GB. This is a serious company knowledge base or a multi-source RAG.
Tens of millions, or high-dimension vectors — now you're at 48–80 GB, and past that across several boxes. Large document estates, multi-tenant retrieval, or you're keeping several indexes hot at once.

A multi-agent system that also holds a big index stacks both costs on the same box — that's how a 32 GB plan turns into a 64 GB one quietly.

The engine choice, briefly

If you already run Postgres, pgvector is the least-effort option — it's an extension, not a new service to babysit. When you have millions of vectors and want fast filtered search, a dedicated engine like Qdrant or Weaviate earns the separate process. Don't over-engineer it on day one; run what you already operate and split it out when search actually slows.

Why bother self-hosting

Two reasons people actually do this, and neither is "to save a few dollars":

Privacy. Embeddings aren't abstract — they encode the text they came from. Your docs, your customers' content, your internal notes, turned into vectors and shipped to a third party's servers. Self-hosting keeps that on a machine you control. If the data is sensitive enough that you're also paying in crypto with no KYC, a managed vector cloud undoes the whole point.

Flat cost. Managed vector services bill by vectors stored and queries run. A VPS is one monthly number and you can hammer it as hard as you like. At scale, predictable beats metered.

What this means for sizing

Start by measuring your corpus, not by guessing. Get your embedding count and dimension, load a sample, watch the resident memory, extrapolate. Then pick a plan with headroom for the index plus everything around it — the app, the model client, room to grow.

For anything past a couple million vectors held privately, the Pro line runs 32 to 80 GB with a dedicated IP and nightly backups, which matters when the index is the product and losing it hurts.