Blog
technical · · 8 min read

How to fix CUDA out of memory in llama.cpp

llama.cpp CUDA out of memory? Read the log, stop pinning -ngl, size the context and KV cache, and leave headroom. Eight fixes for current builds.

By Taskin Karlar, Founder & CEO, Sector88 · LinkedIn
llama-cpp oom gpu-memory kv-cache inference

On 9 November 2026 a llama.cpp user filed a bug report that will look familiar to anyone who runs local models. A mixture-of-experts model loaded fine on a 16 GiB card, then the server died on the first long prompt with CUDA error: out of memory. One line above the crash, llama.cpp had already explained why it could not help: n_gpu_layers already set by user to 999 (issue 31974).

That line is the most useful thing to know about llama.cpp CUDA out of memory errors in late 2026. Current builds size themselves to the GPU. Most crashes now come from flags copied out of older guides that switch that sizing off.

llama.cpp CUDA out of memory is the error you get when the weights, KV cache and compute buffers you asked for need more GPU memory than is free on the device. Here is how to read the error, why the automatic fit does not always save you, and the order in which to fix it.

What does the llama.cpp CUDA out of memory error look like?

It shows up in three places, and each one points at a different buffer.

Where it failsWhat the log saysWhat is usually too bigFirst fix
Loading the modelcudaMalloc failed: out of memory, then unable to allocate CUDA0 bufferThe weights placed on the GPURemove a pinned -ngl, or lower it
Creating the contextfailed to allocate compute pp buffers or failed to initialize the contextKV cache or compute buffersLower -c, quantize the cache, lower -ub
During a requestCUDA error: out of memory with current device: 0, in function ...A buffer that grew after startupLeave more headroom, lower -ub

The load error comes from the CUDA allocator in ggml. It prints the size it asked for, in this form:

allocating <size> MiB on device 0: cudaMalloc failed: out of memory

That number tells you how big the failed request was. Read the first error line, not the last one. Everything after it is fallout.

Why does llama.cpp still run out of memory when fit is on?

Because fit only changes what you did not set. Since pull request 16653 merged on 15 December 2025, llama.cpp checks its projected memory use against free device memory before it allocates anything. If the model will not fit, it shrinks the context first, then moves weights into system RAM.

The defaults, from the server README as of build b12233 (9 November 2026):

  • --fit is on.
  • --fit-target leaves 1,024 MiB free on each GPU.
  • --fit-ctx will not shrink the context below 4,096 tokens.
  • -c defaults to 0, which means the model’s own trained context.
  • -ngl defaults to auto.

The catch is in the same pull request: if you set the context size yourself, it is not changed, and if you set the number of GPU layers, a tensor split or tensor overrides, weight placement is not changed. A follow-up fix made the default layer count a negative number, so that typing -ngl 999 is now recognized as a manual choice.

So these flags, all common in older guides, each switch off part of the safety net:

  1. -ngl 99 or -ngl 999. Fit will no longer move layers off the GPU.
  2. -c 131072, or -c 0 typed explicitly. Fit will no longer shrink the context. Passing -c 0 on purpose asks for the full context and turns off context reduction.
  3. --tensor-split, --override-tensor, --cpu-moe and --n-cpu-moe. The expert offload flags are tensor overrides under the hood, so they also stop fit from placing weights.

The issue that opened this post had all three: -ngl 999, --n-cpu-moe 24 and a 400,000 token context. Fit had nothing left it was allowed to change.

Every flag you set is a promise llama.cpp will not break to save you.

How do you fix llama.cpp CUDA out of memory?

Work down this list in order. Stop when the error goes away.

1. Find out how much you are over

Check what is actually free before llama.cpp starts:

nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free \
           --format=csv,noheader,nounits

Then read llama.cpp’s own numbers. At startup it logs a KV buffer size and a compute buffer size per device. When llama-server exits it prints a memory breakdown table with model, context and compute columns for each GPU and for host memory. Whichever column is largest is the one to cut.

2. Delete the pinned flags and let fit work

Strip -ngl, --tensor-split and --override-tensor from your command and try again. To see what llama.cpp would choose without starting a server, run the fit tool on the same model:

llama-fit-params --model /models/model-Q4_K_M.gguf | tee fit-args.txt
cat fit-args.txt | xargs llama-server --model /models/model-Q4_K_M.gguf

The fit-params README shows this on an RTX 4090 with 24,077 MiB free: a model projected to need 61,807 MiB was fitted by cutting the context from 40,960 to 4,096 tokens and putting all 48 layers on the GPU, with the expert weights of 34 of them moved to system RAM. Treat the printed arguments as a starting point and pin them in your service file once you trust them.

3. Set the context you actually need

If your prompts and replies never pass 8,000 tokens, do not reserve 128,000. Pass -c 8192 or -c 16384. Setting -c turns off fit’s context reduction, so pick a number that fits with room to spare. The arithmetic is in the next section.

4. Quantize the KV cache

llama-server -m /models/model-Q4_K_M.gguf -c 32768 -ctk q8_0 -ctv q8_0

An 8-bit cache uses about half the memory of the default 16-bit cache. A quantized V cache needs flash attention. With -fa left on its default of auto, llama.cpp turns flash attention on for you. With -fa off it refuses to start.

5. Shrink the compute buffers

Compute buffers are reserved from the batch sizes: -b (logical batch, default 2,048) and -ub (physical batch, default 512). The issue above ran -b 4096 -ub 4096, eight times the default physical batch. Go back to the defaults unless you have measured a reason to raise them. Embedding servers are the exception: they need -ub at least as large as the longest input.

6. Leave headroom for anything else on the GPU

Fit measures free memory once, at startup. A desktop session, a second model or a monitoring agent that grabs memory later is not in its sums. Raise the margin with -fitt 2048, or give llama.cpp a GPU of its own with --device or CUDA_VISIBLE_DEVICES.

7. Move weights to system RAM on purpose

When the model genuinely does not fit, choose what leaves the GPU:

  • Mixture-of-experts models. --n-cpu-moe N keeps the expert weights of the first N layers in system RAM, and --cpu-moe keeps all of them there. The dense parts stay on the GPU, which is the cheaper trade.
  • Dense models. Lower -ngl below the layer count, or use --n-cpu-ffn N to keep only the feed-forward weights of the first N layers in system RAM.
  • KV cache. -nkvo keeps the whole cache in system RAM. It frees the most VRAM and costs the most speed.

The first two are manual weight placement, so fit stops managing where weights go once you use them.

8. Prove it at peak

Send your longest real prompt at your real concurrency for a few minutes while you watch nvidia-smi. A server that only works on short prompts will fail on the first long document.

If none of this is enough, the model is too big for the card at that precision. Use a smaller quantization or a smaller model.

How much memory does the KV cache need in llama.cpp?

The KV cache needs 2 x layers x KV heads x head size x bytes per value, for every token of context. It does not depend on the weight quantization, so a 4-bit model with a long context can still run out of memory.

Take Llama 3.1 8B: 32 layers, 8 KV heads and a head size of 128, with a 131,072 token trained context (model card). In llama.cpp’s cache types, q8_0 stores 32 values in 34 bytes and q4_0 stores 32 values in 18 bytes. Our arithmetic:

Context (tokens)f16 cacheq8_0 cacheq4_0 cache
Per token128 KiB68 KiB36 KiB
8,1921 GiB544 MiB288 MiB
32,7684 GiB2.1 GiB1.1 GiB
131,07216 GiB8.5 GiB4.5 GiB

That last row is why -c 131072 copied from a model card fails on a 16 GiB card before a single weight is counted. The GPU memory planning guide has the full budget for weights, cache and activations.

Why does llama-server crash mid-run with CUDA error: out of memory?

Because some GPU buffers grow after startup, and when they cannot, the server stops. Two public reports show the pattern:

  • First long prompt. In issue 31974, opened 9 November 2026, the model sat at about 16,026 MiB of a 16,384 MiB card after loading. The first prompt of about 12,000 tokens needed more workspace and the process aborted.
  • After days of uptime. In issue 28338, a server aborted during an ordinary request after about 76 hours. It was closed as not planned on 21 October 2026.

In both reports the whole process stopped rather than failing one request. Plan for that. Leave more than the default 1,024 MiB margin on a busy card, keep -ub modest, and run the server under a supervisor that restarts it and records why it stopped. If the card is close to full after loading, it is too full.

What should you change first?

SymptomChange firstThen
Fails while loadingRemove -ngl, -ts, -otSmaller quantization
Fails creating the contextLower -c-ctk q8_0 -ctv q8_0
Fails on the first long promptLower -ub and -bRaise -fitt
Fails after hours or daysRaise -fitt, dedicate the GPURestart under a supervisor
Log says “already set by user”Remove that flagRun llama-fit-params

The same logic applies to vLLM, with different flags. If you run both engines, the vLLM OOM guide covers the other half, and the offline setup guide covers getting either one running with no network.

Where Sector88 fits

Memory is the first wall every site hits. Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. It tiers model weights and cache across VRAM and system RAM so larger models fit constrained machines, keeps serving when the network is gone, and keeps a durable record of what ran. See how the platform works, or talk to our engineers about the model and GPU you are trying to fit.

Frequently asked questions

Why does llama.cpp run out of memory when nvidia-smi shows free VRAM?
nvidia-smi shows what is free right now. llama.cpp needs one contiguous allocation of a particular size for each buffer, and some buffers grow after startup during long prompts. If a desktop, another model or a monitoring tool takes memory after llama.cpp has sized itself, the next allocation can fail even though the total looks fine.
Does the fit option fix llama.cpp out of memory automatically?
Only for settings you leave unset. The fit option is on by default. It shrinks the context first, down to 4,096 tokens, then moves weights to system RAM, leaving a 1,024 MiB margin per GPU. If you set the context, the GPU layer count, a tensor split or a tensor override yourself, it leaves those choices alone.
How much VRAM does the KV cache use in llama.cpp?
Two times the number of layers, times the key value heads, times the head size, times the bytes per value, for every token of context. Llama 3.1 8B needs 128 KiB per token at 16-bit. A full 131,072 token context is 16 GiB. With an 8-bit cache it is 8.5 GiB.
Should I still use -ngl 99 in llama.cpp?
Usually not on current builds. The default is auto, which lets llama.cpp work out how many layers fit. Any number you pass, including 99 or 999, counts as a manual choice, so the automatic fit step stops adjusting where the weights go. Use -ngl only when you have measured the right number.
Does quantizing the KV cache hurt output quality?
It trades some precision for memory. An 8-bit cache uses about half the memory of a 16-bit cache and a 4-bit cache about a quarter. Test it on your own prompts before you rely on it, starting with 8-bit, and only drop to 4-bit if 8-bit still does not fit.

Sources

  1. llama.cpp HTTP server README · ggml-org
  2. fit-params README · ggml-org
  3. llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization (pull request 16653) · llama.cpp on GitHub
  4. llama: fix magic number of 999 for GPU layers (pull request 18266) · llama.cpp on GitHub
  5. server: CUDA VMM-pool OOM during prompt processing aborts the process instead of failing (issue 31974) · llama.cpp on GitHub
  6. CUDA OOM hard-abort in ggml_cuda_op_top_k after long-running llama-server decode (issue 28338) · llama.cpp on GitHub
  7. Llama 3.1 8B Instruct model card · Meta on Hugging Face

Written by

Taskin Karlar

Founder & CEO, Sector88

Founder and CEO of Sector88, the infrastructure for AI beyond the cloud.