Best GPU for running LLMs locally

The only question that matters when choosing a GPU for local inference is whether the model fits in video memory. A model that fits runs at the card's full speed. A model that does not spills into system memory and runs perhaps ten to twenty times slower — the difference between a usable assistant and a demonstration.

The arithmetic is simple. Take the parameter count, multiply by the bytes per parameter for your quantisation (roughly 0.55 bytes at 4-bit, 1.1 at 8-bit, 2 at FP16), then add the KV cache for your context length, which grows with both the model size and how much text you keep in play.

That gives you a floor, not a target. Leave headroom: the operating system takes some, and a context window that grows during a long conversation takes more.

Common questions