In late October I had about a week on a DGX Spark. The goal was simple to say: get vLLM, llama.cpp and our native backend serving on it, first with the network up and then with every outbound connection blocked, and write down what broke.
Most of what broke had nothing to do with the models. It came from two facts about the box. GB10 is an arm64 machine with a Blackwell GPU at compute capability 12.1, and its GPU and CPU share one pool of unified memory. A lot of tooling quietly assumes x86, a discrete card and its own VRAM. This post is the list of errors I hit, the fix for each, and the numbers I measured once it all worked.
How I set it up
Every backend ran in a plain Ubuntu 24.04 container with no system CUDA installed and no network access, set up the same way as our GPU image. That way anything the stack needed had to already be inside it. For the offline runs I blocked the container’s outbound traffic, so a backend that tried to download a model, a wheel or a tokenizer would fail on the spot rather than quietly fetch it.
The software pins for the arm64 GPU build were torch 2.10.0 with CUDA 12.8 builds, vLLM 0.19.1 and DCGM for CUDA 12, taken from NVIDIA’s arm64 repo. The CPU side used the same 72 Python packages as our x86 CPU lock, with only torch different.
Why does llama.cpp fail on GB10 with a compute 12.0 build?
This was the first wall. A llama.cpp CUDA build made for Blackwell at 12.0 does not run on GB10.
The reason is how the build names its targets. It compiles 12.x architectures as 12.xa, and those kernels only run on that exact GPU. GB10 is 12.1, so a 12.0 build has nothing it can load. The fix was to build for 121-real on a CUDA 12.9 builder. The resulting wheel then ran fine against the CUDA 12.8 libraries in the lock, so I did not have to move the whole image to 12.9.
The same mistake was hiding in our own checks. Our image check counted a 12.0 build as covering a 12.1 GPU, which would have told an operator the image fits when it does not. It now treats them as different, and it only suggests build profiles for the host’s own platform, so an arm64 box is never offered an x86 image.
Why does the GPU report no memory on unified memory?
On a discrete card you ask the driver how much memory the GPU has and how much is free. On GB10 the driver often does not report it, because there is no separate VRAM to report.
Anything that reads that number then breaks in odd ways. I found 19 places in our code that read GPU memory. All of them now fall back to system memory when the driver reports none. Two related readings needed the same treatment:
- Power limit. GB10 does not report one. Our code was filling in a 250 W default, which was simply wrong. It now reads 0.0 when the GPU does not report a limit, and power draw is read without it. At our defaults the box drew 51 W.
- PCIe link. There is no PCIe link to the GPU in the usual sense, so a link health check raised a false “PCIe degraded” alarm. That alarm is now skipped on unified-memory GPUs.
A small one in the same family: the CPU showed up as “Unknown CPU”. On the Spark it now reads “Cortex-X925 + Cortex-A725”. And the control daemon looked for CUDA libraries only in the x86_64 multiarch folder, so on aarch64 it could not find them. It now uses the host’s own architecture.
After these fixes, 0 of the 6 GPU warnings I had been seeing on GB10 appeared.
Why does vLLM crash on GB10 with free memory?
This is the one that cost the most time, and it is a cousin of the usual vLLM out of memory problem.
vLLM reserves a share of GPU memory at start, set by gpu_memory_utilization. On a discrete card free VRAM is fairly stable. On unified memory the GPU shares the pool with the operating system, the containers and everything else, so free memory moves around.
Our auto-adjust for that share had a floor of 0.5. Once free memory dropped below half the total, vLLM was asked for more than existed and refused to start. With 46.6 of 121.7 GB free, it crashed five times in a row. I made three changes:
- Never ask for more than is free. The 0.5 floor is gone. The share is sized from what is actually free.
- Keep a margin on unified memory. The auto-adjust leaves 4 GB for the rest of the system. At default settings it moved 0.9 down to 0.53 and vLLM served.
- Say what is wrong. vLLM’s two “does not fit” errors now come back as out of memory, with advice to free the GPU or cut the context. The warning also says up front when the model’s weights will not fit the share it leaves.
One more thing to watch: in one session about 23 GB of system memory was not returned after a vLLM test, even with the box idle. On a unified-memory box that memory is gone for everything, not just the GPU, so it is worth checking free memory before you start the next backend.
Two smaller errors worth knowing
A missing C compiler. vLLM needs a C compiler at run time on this setup. In a minimal container it is not there, and the failure did not say so. It is now reported as a dependency error with the package to install.
“pull access denied” from compose. Our GPU and multi-model compose files were pinned to linux/amd64. On the Spark, compose treated the arm64 GPU image that was already loaded as missing, tried to pull it, and failed with “pull access denied”. Removing the pin fixed it. If you see that error on an arm64 box with a local image, check for a platform pin first.
What speeds did I measure?
These are quick serve checks at our default settings, one per backend, not load tests. The soak further down is the number under load.
| Run | llama.cpp CUDA | vLLM | Native |
|---|---|---|---|
| First bring up, Qwen2.5-0.5B (Q4_K_M for llama.cpp), no network | 233.5 tok/s | served, correct answer | 122.6 tok/s |
| GPU image built on the Spark, offline | 246 tok/s | 162 tok/s | 122 tok/s |
| Published GPU image, online | 246 tok/s | 152 tok/s | 111 tok/s |
| Published GPU image, outbound blocked | 223 tok/s | answered | answered |
On the CPU side, with no GPU involved:
- The arm64 CPU image on the Spark’s CPU, no network: 90 to 97 tok/s.
- An aarch64 offline bundle installed on a bare host with no network: 121 to 125 tok/s, with 0 GPU warnings.
The 6 hour soak
The quick checks tell you a backend works. A soak tells you it keeps working. I ran llama.cpp CUDA with Qwen2.5-7B Q4_K_M and 4 clients for 6 hours.
| Metric | Result |
|---|---|
| Requests | 3,789, 0 errors |
| Tokens generated | 943,531 |
| Throughput | 43.7 tok/s average, 43.6 tok/s in both the first and last 10 minutes |
| Temperature | 72 °C average, 75 °C peak |
| Power | 51 W average, no throttling |
The first and last ten minutes matching is the part I care about most. It means no slow drift from heat or memory over the run. Over a weekend I also left a lighter run going for 64 hours: 3,497 requests, 0 errors.
A checklist if you are doing this yourself
- Build llama.cpp for
121-real, not 120. Check your image’s own architecture check too. - Treat “no GPU memory reported” as normal on unified memory and fall back to system memory everywhere you read it.
- Size
gpu_memory_utilizationfrom free memory with a margin. Do not use a fixed floor. - Check free system memory between backend runs.
- Install a C compiler in the image if vLLM needs one.
- Remove
linux/amd64pins from compose files that should run on arm64. - Test offline with outbound traffic actually blocked, not just unplugged in your head. That is how you find the download you forgot about.
Where Sector88 fits
Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off. Part of that job is making each new class of hardware boring: the errors above get found and fixed once, so an operator gets a stack that serves on the first try, with the network up or blocked.