On 9 November 2026 a llama.cpp user filed a bug report that will look familiar to anyone who runs local models. A mixture-of-experts model loaded fine on a 16 GiB card, then the server died on the first long prompt with CUDA error: out of memory. One line above the crash, llama.cpp had already explained why it could not help: n_gpu_layers already set by user to 999 (issue 31974).
That line is the most useful thing to know about llama.cpp CUDA out of memory errors in late 2026. Current builds size themselves to the GPU. Most crashes now come from flags copied out of older guides that switch that sizing off.
llama.cpp CUDA out of memory is the error you get when the weights, KV cache and compute buffers you asked for need more GPU memory than is free on the device. Here is how to read the error, why the automatic fit does not always save you, and the order in which to fix it.
What does the llama.cpp CUDA out of memory error look like?
It shows up in three places, and each one points at a different buffer.
| Where it fails | What the log says | What is usually too big | First fix |
|---|---|---|---|
| Loading the model | cudaMalloc failed: out of memory, then unable to allocate CUDA0 buffer | The weights placed on the GPU | Remove a pinned -ngl, or lower it |
| Creating the context | failed to allocate compute pp buffers or failed to initialize the context | KV cache or compute buffers | Lower -c, quantize the cache, lower -ub |
| During a request | CUDA error: out of memory with current device: 0, in function ... | A buffer that grew after startup | Leave more headroom, lower -ub |
The load error comes from the CUDA allocator in ggml. It prints the size it asked for, in this form:
allocating <size> MiB on device 0: cudaMalloc failed: out of memory
That number tells you how big the failed request was. Read the first error line, not the last one. Everything after it is fallout.
Why does llama.cpp still run out of memory when fit is on?
Because fit only changes what you did not set. Since pull request 16653 merged on 15 December 2025, llama.cpp checks its projected memory use against free device memory before it allocates anything. If the model will not fit, it shrinks the context first, then moves weights into system RAM.
The defaults, from the server README as of build b12233 (9 November 2026):
--fitison.--fit-targetleaves 1,024 MiB free on each GPU.--fit-ctxwill not shrink the context below 4,096 tokens.-cdefaults to 0, which means the model’s own trained context.-ngldefaults toauto.
The catch is in the same pull request: if you set the context size yourself, it is not changed, and if you set the number of GPU layers, a tensor split or tensor overrides, weight placement is not changed. A follow-up fix made the default layer count a negative number, so that typing -ngl 999 is now recognized as a manual choice.
So these flags, all common in older guides, each switch off part of the safety net:
-ngl 99or-ngl 999. Fit will no longer move layers off the GPU.-c 131072, or-c 0typed explicitly. Fit will no longer shrink the context. Passing-c 0on purpose asks for the full context and turns off context reduction.--tensor-split,--override-tensor,--cpu-moeand--n-cpu-moe. The expert offload flags are tensor overrides under the hood, so they also stop fit from placing weights.
The issue that opened this post had all three: -ngl 999, --n-cpu-moe 24 and a 400,000 token context. Fit had nothing left it was allowed to change.
Every flag you set is a promise llama.cpp will not break to save you.
How do you fix llama.cpp CUDA out of memory?
Work down this list in order. Stop when the error goes away.
1. Find out how much you are over
Check what is actually free before llama.cpp starts:
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free \
--format=csv,noheader,nounits
Then read llama.cpp’s own numbers. At startup it logs a KV buffer size and a compute buffer size per device. When llama-server exits it prints a memory breakdown table with model, context and compute columns for each GPU and for host memory. Whichever column is largest is the one to cut.
2. Delete the pinned flags and let fit work
Strip -ngl, --tensor-split and --override-tensor from your command and try again. To see what llama.cpp would choose without starting a server, run the fit tool on the same model:
llama-fit-params --model /models/model-Q4_K_M.gguf | tee fit-args.txt
cat fit-args.txt | xargs llama-server --model /models/model-Q4_K_M.gguf
The fit-params README shows this on an RTX 4090 with 24,077 MiB free: a model projected to need 61,807 MiB was fitted by cutting the context from 40,960 to 4,096 tokens and putting all 48 layers on the GPU, with the expert weights of 34 of them moved to system RAM. Treat the printed arguments as a starting point and pin them in your service file once you trust them.
3. Set the context you actually need
If your prompts and replies never pass 8,000 tokens, do not reserve 128,000. Pass -c 8192 or -c 16384. Setting -c turns off fit’s context reduction, so pick a number that fits with room to spare. The arithmetic is in the next section.
4. Quantize the KV cache
llama-server -m /models/model-Q4_K_M.gguf -c 32768 -ctk q8_0 -ctv q8_0
An 8-bit cache uses about half the memory of the default 16-bit cache. A quantized V cache needs flash attention. With -fa left on its default of auto, llama.cpp turns flash attention on for you. With -fa off it refuses to start.
5. Shrink the compute buffers
Compute buffers are reserved from the batch sizes: -b (logical batch, default 2,048) and -ub (physical batch, default 512). The issue above ran -b 4096 -ub 4096, eight times the default physical batch. Go back to the defaults unless you have measured a reason to raise them. Embedding servers are the exception: they need -ub at least as large as the longest input.
6. Leave headroom for anything else on the GPU
Fit measures free memory once, at startup. A desktop session, a second model or a monitoring agent that grabs memory later is not in its sums. Raise the margin with -fitt 2048, or give llama.cpp a GPU of its own with --device or CUDA_VISIBLE_DEVICES.
7. Move weights to system RAM on purpose
When the model genuinely does not fit, choose what leaves the GPU:
- Mixture-of-experts models.
--n-cpu-moe Nkeeps the expert weights of the first N layers in system RAM, and--cpu-moekeeps all of them there. The dense parts stay on the GPU, which is the cheaper trade. - Dense models. Lower
-nglbelow the layer count, or use--n-cpu-ffn Nto keep only the feed-forward weights of the first N layers in system RAM. - KV cache.
-nkvokeeps the whole cache in system RAM. It frees the most VRAM and costs the most speed.
The first two are manual weight placement, so fit stops managing where weights go once you use them.
8. Prove it at peak
Send your longest real prompt at your real concurrency for a few minutes while you watch nvidia-smi. A server that only works on short prompts will fail on the first long document.
If none of this is enough, the model is too big for the card at that precision. Use a smaller quantization or a smaller model.
How much memory does the KV cache need in llama.cpp?
The KV cache needs 2 x layers x KV heads x head size x bytes per value, for every token of context. It does not depend on the weight quantization, so a 4-bit model with a long context can still run out of memory.
Take Llama 3.1 8B: 32 layers, 8 KV heads and a head size of 128, with a 131,072 token trained context (model card). In llama.cpp’s cache types, q8_0 stores 32 values in 34 bytes and q4_0 stores 32 values in 18 bytes. Our arithmetic:
| Context (tokens) | f16 cache | q8_0 cache | q4_0 cache |
|---|---|---|---|
| Per token | 128 KiB | 68 KiB | 36 KiB |
| 8,192 | 1 GiB | 544 MiB | 288 MiB |
| 32,768 | 4 GiB | 2.1 GiB | 1.1 GiB |
| 131,072 | 16 GiB | 8.5 GiB | 4.5 GiB |
That last row is why -c 131072 copied from a model card fails on a 16 GiB card before a single weight is counted. The GPU memory planning guide has the full budget for weights, cache and activations.
Why does llama-server crash mid-run with CUDA error: out of memory?
Because some GPU buffers grow after startup, and when they cannot, the server stops. Two public reports show the pattern:
- First long prompt. In issue 31974, opened 9 November 2026, the model sat at about 16,026 MiB of a 16,384 MiB card after loading. The first prompt of about 12,000 tokens needed more workspace and the process aborted.
- After days of uptime. In issue 28338, a server aborted during an ordinary request after about 76 hours. It was closed as not planned on 21 October 2026.
In both reports the whole process stopped rather than failing one request. Plan for that. Leave more than the default 1,024 MiB margin on a busy card, keep -ub modest, and run the server under a supervisor that restarts it and records why it stopped. If the card is close to full after loading, it is too full.
What should you change first?
| Symptom | Change first | Then |
|---|---|---|
| Fails while loading | Remove -ngl, -ts, -ot | Smaller quantization |
| Fails creating the context | Lower -c | -ctk q8_0 -ctv q8_0 |
| Fails on the first long prompt | Lower -ub and -b | Raise -fitt |
| Fails after hours or days | Raise -fitt, dedicate the GPU | Restart under a supervisor |
| Log says “already set by user” | Remove that flag | Run llama-fit-params |
The same logic applies to vLLM, with different flags. If you run both engines, the vLLM OOM guide covers the other half, and the offline setup guide covers getting either one running with no network.
Where Sector88 fits
Memory is the first wall every site hits. Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. It tiers model weights and cache across VRAM and system RAM so larger models fit constrained machines, keeps serving when the network is gone, and keeps a durable record of what ran. See how the platform works, or talk to our engineers about the model and GPU you are trying to fit.