How much VRAM do you need for an AI model?

Every local AI setup comes down to one question: does the model fit in video memory? Here are the figures, before context.

A 7B model needs roughly 4GB at 4-bit and 8GB at 8-bit, so plan on 8GB. A 13B model needs roughly 8GB at 4-bit and 16GB at 8-bit, so plan on 16GB. A 32B model needs roughly 20GB at 4-bit and 36GB at 8-bit, so plan on 24GB. A 70B model needs roughly 40GB at 4-bit and 75GB at 8-bit, so plan on 48GB across two cards. A 120B model needs roughly 68GB at 4-bit and 130GB at 8-bit, so plan on 128GB across four cards.

Add the KV cache on top. At long context lengths it is not a rounding error — a 32k context on a 70B model can add several gigabytes. If a configuration only just fits on paper, it will not fit in practice.