# How to fix CUDA out of memory in llama.cpp

> llama.cpp CUDA out of memory? Read the log, stop pinning -ngl, size the context and KV cache, and leave headroom. Eight fixes for current builds.

- URL: https://www.sector88.co/blog/llama-cpp-cuda-out-of-memory-fix
- Author: Taskin Karlar
- Published: 2026-11-10
- Updated: 2026-11-10
- Category: Technical
- Tags: llama-cpp, oom, gpu-memory, kv-cache, inference

## Quick answer

llama.cpp runs out of CUDA memory when the weights, the KV cache and the compute buffers you asked for need more GPU memory than is free. Recent builds size themselves to the GPU automatically, but only for settings you leave unset. Remove pinned layer counts, set a realistic context length, quantize the KV cache and leave headroom.

On 9 November 2026 a llama.cpp user filed a bug report that will look familiar to anyone who runs local models. A mixture-of-experts model loaded fine on a 16 GiB card, then the server died on the first long prompt with `CUDA error: out of memory`. One line above the crash, llama.cpp had already explained why it could not help: `n_gpu_layers already set by user to 999` ([issue 31974](https://github.com/ggml-org/llama.cpp/issues/31974)).

That line is the most useful thing to know about llama.cpp CUDA out of memory errors in late 2026. Current builds size themselves to the GPU. Most crashes now come from flags copied out of older guides that switch that sizing off.

**llama.cpp CUDA out of memory is the error you get when the weights, KV cache and compute buffers you asked for need more GPU memory than is free on the device.** Here is how to read the error, why the automatic fit does not always save you, and the order in which to fix it.

## What does the llama.cpp CUDA out of memory error look like?

It shows up in three places, and each one points at a different buffer.

| Where it fails | What the log says | What is usually too big | First fix |
|---|---|---|---|
| Loading the model | `cudaMalloc failed: out of memory`, then `unable to allocate CUDA0 buffer` | The weights placed on the GPU | Remove a pinned `-ngl`, or lower it |
| Creating the context | `failed to allocate compute pp buffers` or `failed to initialize the context` | KV cache or compute buffers | Lower `-c`, quantize the cache, lower `-ub` |
| During a request | `CUDA error: out of memory` with `current device: 0, in function ...` | A buffer that grew after startup | Leave more headroom, lower `-ub` |

The load error comes from the CUDA allocator in ggml. It prints the size it asked for, in this form:

```text
allocating <size> MiB on device 0: cudaMalloc failed: out of memory
```

That number tells you how big the failed request was. Read the first error line, not the last one. Everything after it is fallout.

## Why does llama.cpp still run out of memory when fit is on?

Because fit only changes what you did not set. Since [pull request 16653](https://github.com/ggml-org/llama.cpp/pull/16653) merged on 15 December 2025, llama.cpp checks its projected memory use against free device memory before it allocates anything. If the model will not fit, it shrinks the context first, then moves weights into system RAM.

The defaults, from the [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) as of build b12233 (9 November 2026):

- `--fit` is `on`.
- `--fit-target` leaves 1,024 MiB free on each GPU.
- `--fit-ctx` will not shrink the context below 4,096 tokens.
- `-c` defaults to 0, which means the model's own trained context.
- `-ngl` defaults to `auto`.

The catch is in the same pull request: if you set the context size yourself, it is not changed, and if you set the number of GPU layers, a tensor split or tensor overrides, weight placement is not changed. A [follow-up fix](https://github.com/ggml-org/llama.cpp/pull/18266) made the default layer count a negative number, so that typing `-ngl 999` is now recognized as a manual choice.

So these flags, all common in older guides, each switch off part of the safety net:

1. **`-ngl 99` or `-ngl 999`.** Fit will no longer move layers off the GPU.
2. **`-c 131072`, or `-c 0` typed explicitly.** Fit will no longer shrink the context. Passing `-c 0` on purpose asks for the full context and turns off context reduction.
3. **`--tensor-split`, `--override-tensor`, `--cpu-moe` and `--n-cpu-moe`.** The expert offload flags are tensor overrides under the hood, so they also stop fit from placing weights.

The issue that opened this post had all three: `-ngl 999`, `--n-cpu-moe 24` and a 400,000 token context. Fit had nothing left it was allowed to change.

Every flag you set is a promise llama.cpp will not break to save you.

## How do you fix llama.cpp CUDA out of memory?

Work down this list in order. Stop when the error goes away.

### 1. Find out how much you are over

Check what is actually free before llama.cpp starts:

```bash
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free \
           --format=csv,noheader,nounits
```

Then read llama.cpp's own numbers. At startup it logs a `KV buffer size` and a `compute buffer size` per device. When `llama-server` exits it prints a memory breakdown table with model, context and compute columns for each GPU and for host memory. Whichever column is largest is the one to cut.

### 2. Delete the pinned flags and let fit work

Strip `-ngl`, `--tensor-split` and `--override-tensor` from your command and try again. To see what llama.cpp would choose without starting a server, run the fit tool on the same model:

```bash
llama-fit-params --model /models/model-Q4_K_M.gguf | tee fit-args.txt
cat fit-args.txt | xargs llama-server --model /models/model-Q4_K_M.gguf
```

The [fit-params README](https://github.com/ggml-org/llama.cpp/blob/master/tools/fit-params/README.md) shows this on an RTX 4090 with 24,077 MiB free: a model projected to need 61,807 MiB was fitted by cutting the context from 40,960 to 4,096 tokens and putting all 48 layers on the GPU, with the expert weights of 34 of them moved to system RAM. Treat the printed arguments as a starting point and pin them in your service file once you trust them.

### 3. Set the context you actually need

If your prompts and replies never pass 8,000 tokens, do not reserve 128,000. Pass `-c 8192` or `-c 16384`. Setting `-c` turns off fit's context reduction, so pick a number that fits with room to spare. The arithmetic is in the next section.

### 4. Quantize the KV cache

```bash
llama-server -m /models/model-Q4_K_M.gguf -c 32768 -ctk q8_0 -ctv q8_0
```

An 8-bit cache uses about half the memory of the default 16-bit cache. A quantized V cache needs flash attention. With `-fa` left on its default of `auto`, llama.cpp turns flash attention on for you. With `-fa off` it refuses to start.

### 5. Shrink the compute buffers

Compute buffers are reserved from the batch sizes: `-b` (logical batch, default 2,048) and `-ub` (physical batch, default 512). The issue above ran `-b 4096 -ub 4096`, eight times the default physical batch. Go back to the defaults unless you have measured a reason to raise them. Embedding servers are the exception: they need `-ub` at least as large as the longest input.

### 6. Leave headroom for anything else on the GPU

Fit measures free memory once, at startup. A desktop session, a second model or a monitoring agent that grabs memory later is not in its sums. Raise the margin with `-fitt 2048`, or give llama.cpp a GPU of its own with `--device` or `CUDA_VISIBLE_DEVICES`.

### 7. Move weights to system RAM on purpose

When the model genuinely does not fit, choose what leaves the GPU:

- **Mixture-of-experts models.** `--n-cpu-moe N` keeps the expert weights of the first N layers in system RAM, and `--cpu-moe` keeps all of them there. The dense parts stay on the GPU, which is the cheaper trade.
- **Dense models.** Lower `-ngl` below the layer count, or use `--n-cpu-ffn N` to keep only the feed-forward weights of the first N layers in system RAM.
- **KV cache.** `-nkvo` keeps the whole cache in system RAM. It frees the most VRAM and costs the most speed.

The first two are manual weight placement, so fit stops managing where weights go once you use them.

### 8. Prove it at peak

Send your longest real prompt at your real concurrency for a few minutes while you watch `nvidia-smi`. A server that only works on short prompts will fail on the first long document.

If none of this is enough, the model is too big for the card at that precision. Use a smaller quantization or a smaller model.

## How much memory does the KV cache need in llama.cpp?

**The KV cache needs 2 x layers x KV heads x head size x bytes per value, for every token of context.** It does not depend on the weight quantization, so a 4-bit model with a long context can still run out of memory.

Take Llama 3.1 8B: 32 layers, 8 KV heads and a head size of 128, with a 131,072 token trained context ([model card](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct)). In llama.cpp's cache types, q8_0 stores 32 values in 34 bytes and q4_0 stores 32 values in 18 bytes. Our arithmetic:

| Context (tokens) | f16 cache | q8_0 cache | q4_0 cache |
|---|---|---|---|
| Per token | 128 KiB | 68 KiB | 36 KiB |
| 8,192 | 1 GiB | 544 MiB | 288 MiB |
| 32,768 | 4 GiB | 2.1 GiB | 1.1 GiB |
| 131,072 | 16 GiB | 8.5 GiB | 4.5 GiB |

That last row is why `-c 131072` copied from a model card fails on a 16 GiB card before a single weight is counted. The [GPU memory planning guide](/blog/gpu-memory-planning-guide-vram-system-ram) has the full budget for weights, cache and activations.

## Why does llama-server crash mid-run with CUDA error: out of memory?

Because some GPU buffers grow after startup, and when they cannot, the server stops. Two public reports show the pattern:

- **First long prompt.** In [issue 31974](https://github.com/ggml-org/llama.cpp/issues/31974), opened 9 November 2026, the model sat at about 16,026 MiB of a 16,384 MiB card after loading. The first prompt of about 12,000 tokens needed more workspace and the process aborted.
- **After days of uptime.** In [issue 28338](https://github.com/ggml-org/llama.cpp/issues/28338), a server aborted during an ordinary request after about 76 hours. It was closed as not planned on 21 October 2026.

In both reports the whole process stopped rather than failing one request. Plan for that. Leave more than the default 1,024 MiB margin on a busy card, keep `-ub` modest, and run the server under a supervisor that restarts it and records why it stopped. If the card is close to full after loading, it is too full.

## What should you change first?

| Symptom | Change first | Then |
|---|---|---|
| Fails while loading | Remove `-ngl`, `-ts`, `-ot` | Smaller quantization |
| Fails creating the context | Lower `-c` | `-ctk q8_0 -ctv q8_0` |
| Fails on the first long prompt | Lower `-ub` and `-b` | Raise `-fitt` |
| Fails after hours or days | Raise `-fitt`, dedicate the GPU | Restart under a supervisor |
| Log says "already set by user" | Remove that flag | Run `llama-fit-params` |

The same logic applies to vLLM, with different flags. If you run both engines, the [vLLM OOM guide](/blog/how-to-fix-vllm-oom) covers the other half, and the [offline setup guide](/blog/run-vllm-llama-cpp-offline-air-gapped) covers getting either one running with no network.

## Where Sector88 fits

Memory is the first wall every site hits. Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. It tiers model weights and cache across VRAM and system RAM so larger models fit constrained machines, keeps serving when the network is gone, and keeps a durable record of what ran. See [how the platform works](/platform), or [talk to our engineers](/contact) about the model and GPU you are trying to fit.

## Frequently asked questions

### Why does llama.cpp run out of memory when nvidia-smi shows free VRAM?

nvidia-smi shows what is free right now. llama.cpp needs one contiguous allocation of a particular size for each buffer, and some buffers grow after startup during long prompts. If a desktop, another model or a monitoring tool takes memory after llama.cpp has sized itself, the next allocation can fail even though the total looks fine.

### Does the fit option fix llama.cpp out of memory automatically?

Only for settings you leave unset. The fit option is on by default. It shrinks the context first, down to 4,096 tokens, then moves weights to system RAM, leaving a 1,024 MiB margin per GPU. If you set the context, the GPU layer count, a tensor split or a tensor override yourself, it leaves those choices alone.

### How much VRAM does the KV cache use in llama.cpp?

Two times the number of layers, times the key value heads, times the head size, times the bytes per value, for every token of context. Llama 3.1 8B needs 128 KiB per token at 16-bit. A full 131,072 token context is 16 GiB. With an 8-bit cache it is 8.5 GiB.

### Should I still use -ngl 99 in llama.cpp?

Usually not on current builds. The default is auto, which lets llama.cpp work out how many layers fit. Any number you pass, including 99 or 999, counts as a manual choice, so the automatic fit step stops adjusting where the weights go. Use -ngl only when you have measured the right number.

### Does quantizing the KV cache hurt output quality?

It trades some precision for memory. An 8-bit cache uses about half the memory of a 16-bit cache and a 4-bit cache about a quarter. Test it on your own prompts before you rely on it, starting with 8-bit, and only drop to 4-bit if 8-bit still does not fit.

## Sources

- [llama.cpp HTTP server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), ggml-org
- [fit-params README](https://github.com/ggml-org/llama.cpp/blob/master/tools/fit-params/README.md), ggml-org
- [llama: automatically set parameters not set by the user in such a way that maximizes GPU utilization (pull request 16653)](https://github.com/ggml-org/llama.cpp/pull/16653), llama.cpp on GitHub
- [llama: fix magic number of 999 for GPU layers (pull request 18266)](https://github.com/ggml-org/llama.cpp/pull/18266), llama.cpp on GitHub
- [server: CUDA VMM-pool OOM during prompt processing aborts the process instead of failing (issue 31974)](https://github.com/ggml-org/llama.cpp/issues/31974), llama.cpp on GitHub
- [CUDA OOM hard-abort in ggml_cuda_op_top_k after long-running llama-server decode (issue 28338)](https://github.com/ggml-org/llama.cpp/issues/28338), llama.cpp on GitHub
- [Llama 3.1 8B Instruct model card](https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct), Meta on Hugging Face
