How much VRAM do I need to run an LLM?

Multiply the parameter count by the bytes per parameter at your precision for the weights, add the KV cache for your peak context and concurrency, and add a margin of roughly 10 to 20 percent for the engine, CUDA context and activations. An 8 billion parameter model at 16-bit needs about 16 GB for weights alone, before any KV cache.

How do I calculate KV cache memory?

KV cache per token is 2 times the number of layers times the number of key value heads times the head dimension times the bytes per element. Multiply by the context length and by the number of sequences served at once. Models with grouped query attention have far fewer key value heads than attention heads, which shrinks the cache.

How much system RAM does a GPU inference server need?

At minimum, enough to hold the operating system, the serving process and a copy of the model while it loads. If you tier memory, add the full size of whatever spills out of VRAM, held in pinned memory. A common starting point is at least twice the total VRAM, then adjust to the tiering plan.

What is memory tiering for AI inference?

Memory tiering places parts of the model weights and KV cache in different memory tiers. The most used data sits in VRAM, the rest in system RAM, and data moves between them as needed. It lets a model run when it does not fit in VRAM alone, at the cost of extra latency when data crosses the PCIe bus.

Does quantisation reduce KV cache memory?

Weight quantisation reduces the weight share only. The KV cache stays in the precision the engine is configured for. Some engines support a quantised KV cache, such as 8-bit, which roughly halves cache memory at a small quality cost that you should check on your own workload.

GPU memory planning: a guide to VRAM and system RAM

Most AI inference failures at a site are memory failures that were decided at purchase. Someone sized the GPU from the model’s parameter count, forgot the KV cache, forgot the second model, forgot the host, and the system that worked in testing falls over on its first busy day.

GPU memory planning is arithmetic, not guesswork. This guide sets out the budget for VRAM, then for system RAM as the second tier, so you can size a machine before you buy it and know why it will hold.

What is GPU memory planning?

GPU memory planning is working out, before deployment, how much VRAM and system RAM a set of models needs at peak load, and where each part of each model will live. It covers model weights, the KV cache, engine overhead and the host itself, at the real context lengths and concurrency the site will see.

Step 1. Budget the weights

Weights are the easy part. Multiply parameters by bytes per parameter.

Precision	Bytes per parameter	8B model	70B model
16-bit (FP16, BF16)	2	about 16 GB	about 140 GB
8-bit (FP8, INT8)	1	about 8 GB	about 70 GB
4-bit (common GGUF, AWQ, GPTQ)	about 0.5 to 0.6	about 4 to 5 GB	about 35 to 42 GB

Four-bit formats carry a little extra for scales and some layers kept at higher precision, which is why the range is wider. Read the actual file sizes for the quantisation you plan to run rather than relying on the formula alone.

Precision is a quality decision as well as a memory one. Evaluate the quantised model on your own workload before you plan around it.

Step 2. Budget the KV cache

The KV cache is where plans go wrong, because it grows with use. Every token in every active sequence holds a key and a value for every layer.

The formula:

KV bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_element
KV total = KV bytes per token x context_length x concurrent_sequences

Take a model with 32 layers, 8 key value heads and a head dimension of 128, which matches the published shape of Llama 3 8B in its model card and configuration. At 16-bit, that is 2 x 32 x 8 x 128 x 2 bytes, or 128 KB per token. An 8,192 token context costs 1 GB per sequence. Sixteen concurrent sequences at full context cost 16 GB, as much as the weights.

Two things shrink this number. Grouped query attention uses fewer key value heads than attention heads, which is why modern models are far cheaper here than older ones. Paged allocation, introduced with PagedAttention, stops engines wasting memory on reserved but unused context. Neither removes the need to plan for peak.

Plan the KV cache from real traffic: the longest prompt plus response you will serve, times the peak number of sequences you will serve at once. Not the model’s maximum context, and not an average.

Step 3. Add the fixed overhead

The engine, the CUDA context, activation buffers and any captured graphs take memory before the first request arrives. A planning margin of 10 to 20 percent of VRAM is a reasonable start. Then measure it on the target machine, because it varies with engine, version and settings.

Engines such as vLLM reserve a share of VRAM at startup through settings like gpu_memory_utilization, documented in the engine arguments. Your plan has to fit inside that share, not inside the card’s nominal size. If a display or another process uses the GPU, subtract that first.

Step 4. Add every model, not just the main one

Sites rarely run one model. A typical set is a main language model, an embedding model for retrieval, perhaps a reranker, perhaps a vision or speech model. Each one needs weights, its own working memory and its own margin.

Write them all down in one table:

Model	Role	Weights	KV or working memory at peak	Margin	Total
Main LLM	Answers and summaries
Embedding model	Retrieval
Reranker	Retrieval
Speech or vision	Input
Site total

If the site total fits in VRAM with margin, you are done with the GPU side. If it does not, you have three choices: a smaller or more heavily quantised model, more VRAM, or a second memory tier.

Step 5. Plan system RAM as the second tier

System RAM is slower than VRAM but far larger and far cheaper per gigabyte. Treating it as a planned tier, rather than an accident, is what lets a model run on a machine where it does not fit VRAM alone.

There are two common ways to use it. Engines can offload part of the weights to system RAM and copy them over PCIe as needed; vLLM does this with --cpu-offload-gb. Layer splitting, as in llama.cpp, runs some layers on the GPU and the rest on the CPU. Tier-aware orchestration goes further and moves weights and KV cache blocks between tiers based on what is being used.

Whichever you choose, budget system RAM for:

The operating system and services, with room to spare.
The serving processes themselves.
Model loading. Many loaders hold a full or partial copy of the weights in system RAM while they transfer to the GPU.
The spill tier. The full size of any weights or KV cache you plan to hold in system RAM.
Pinned memory. Data that moves to and from the GPU often should sit in page-locked host memory so transfers run at full speed. Pinned memory cannot be swapped out, so it comes off the top of what the host can use for anything else.

Do not plan on swap. A model that spills from VRAM into system RAM is slower. A model that spills from system RAM onto swap is broken.

Step 6. Understand the cost of crossing tiers

Every byte that crosses from system RAM to VRAM crosses the PCIe bus, which is an order of magnitude slower than VRAM. That is the trade tiering makes: capacity in exchange for latency.

The practical consequences for planning:

Keep the hottest data in VRAM. Active KV blocks and the layers used on every token belong on the GPU.
Prefer workloads that tolerate latency on the tiered model. Batch summarisation, overnight analysis and document processing absorb tier crossings well. Interactive chat on a heavily tiered model will feel slow.
Check the bus. A GPU in a slot with fewer PCIe lanes than it supports moves tiered data more slowly. Confirm the lane width the slot actually provides.
Measure on your own workload. Throughput on a tiered system depends heavily on model, context and access pattern. Plan from your own measurements, not from someone else’s figures.

A worked plan

Here is the reasoning for a site that needs a main model and an embedding model, written as a sequence rather than a result.

Pick the main model and quantisation that pass your evaluation. Read the real file size.
Set the real context length and peak concurrency from expected traffic. Compute the KV cache with the formula above.
Add the embedding model’s weights and working memory.
Add the engine margin, measured on the target machine.
Compare the total with usable VRAM. If it fits with 10 percent spare, stop.
If it does not, decide what moves to system RAM: usually cold weights first, then older KV blocks. Size system RAM for that spill plus the host, the loaders and pinned buffers.
Load test at peak and watch both tiers. The plan is only real once it survives the busiest hour you expect.

Common mistakes

Sizing from parameters alone and forgetting the KV cache.
Planning for the model’s maximum context instead of real traffic.
Forgetting the second and third models.
Buying a GPU with generous VRAM and a host with too little system RAM to load the model.
Treating swap as a memory tier.
Running another process on the GPU that the plan never counted.

If you are already seeing out of memory errors, our guide to fixing vLLM OOM walks through the settings in order.

Where Sector88 fits

Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. It tiers model weights and KV cache across VRAM and system RAM, so a site can run the models it needs on the memory it has.

The rule that holds for every plan: weights are the number you can look up, the KV cache is the number that breaks you, and system RAM is the tier you should plan rather than discover.