On 2 November 2026 vLLM shipped v0.33.0. Among 927 commits, one line matters to anyone running it at a remote site: release wheels and images are now built for CUDA 13.0 only, and the CUDA 12.9 builds are no longer published (release notes). NVIDIA’s release notes say CUDA 13 applications run on drivers from the 580 branch or newer.
Upgrade a site that runs an older driver, without updating the driver too, and the new build can fail to start. At headquarters you pull the old image again. At a site with no network, there is no again.
That is the whole problem with model updates offline. Rollback only works if the last good release is already on the machine. This guide shows how to package releases so you can roll back an LLM update with one switch and one restart, using vLLM or llama.cpp.
What is a model rollback?
A model rollback is returning a serving machine to the last release that was known to work: the same weights, engine, flags and chat template, byte for byte.
The word “model” hides most of the risk. A new set of weights is only one kind of update. A new engine version, a new quantization file or one changed flag can change answers, memory use or whether the server starts at all. If you can only roll back the weights, you cannot roll back most of what breaks.
What actually changes in a model update?
At least six things, and each needs to be pinned and recorded.
| Part | What can change | How to pin it | Where to read it back |
|---|---|---|---|
| Weights | New fine tune or new base model | Model commit hash, file checksums | Release record, sha256sum -c |
| Quantization | Different GGUF file or precision | Exact file name and checksum | llama.cpp GET /props shows model_path |
| Tokenizer and chat template | Prompt format, stop tokens | Ship the files in the release folder | llama.cpp GET /props shows chat_template |
| Engine | vLLM or llama.cpp version | Image tag or wheel lock, never latest | vLLM GET /version, llama.cpp build_info |
| Driver and CUDA | Which builds can run | Driver branch in the release record | nvidia-smi |
| Flags | Context, memory share, parallel slots | One env file per release | The release folder itself |
The 10 November guide to fixing CUDA out of memory in llama.cpp shows why flags belong in that list. A fitted set of flags is only right for one model, one build and one GPU. Pin it per release, not per machine.
Why is rollback harder at a site with no network?
Because every recovery move you use in the cloud assumes a download.
- You cannot pull the old version. Registries, package indexes and Hugging Face are out of reach. If the old image was deleted to make room, it is gone.
- Upstream may stop publishing it. vLLM’s CUDA 12.9 builds stopped with v0.33.0 (pull request 62575). An old version you never saved may not be available even when you do have a link.
- Nobody may be on site. A fix that needs someone to plug in a laptop can take days.
- The update itself is expensive to move. Weights run to tens of gigabytes and often travel on a drive. You do not want to repeat that trip to undo a mistake.
The rule follows: never delete the last known good release until the new one has proven itself at that site.
How do you package a release you can roll back?
Make each release one self-contained folder, and never edit it after it ships.
/srv/ai/releases/
2026-10-20-qwen3-8b-r4/
model/ # weights, tokenizer, chat template
engine/ # vllm-openai image tar, or the llama-server binary
serve.env # every flag and environment variable
golden.jsonl # canary prompts with expected checks
release.json # IDs, versions, approvals
SHA256SUMS # checksum of every file above
2026-11-12-qwen3-8b-r5/
...
/srv/ai/current -> releases/2026-10-20-qwen3-8b-r4
The offline setup guide covers staging weights, wheels and images with no internet. Three habits from it matter most for rollback:
- Download weights with a pinned
--revision. When you usehf download --local-dir, Hugging Face writes a.cache/huggingface/download/folder of metadata next to the files (download guide). The first line of each.metadatafile is the commit hash. Copy it intorelease.jsonbefore you ship. - Save the engine with the model.
docker savethe exact image tag, or copy the llama-server binary built for that release. The engine is part of the release. - Write the checksum list last. Run
sha256sumover the whole folder, and check it on arrival withsha256sum -c SHA256SUMS.
A minimal release.json looks like this:
{
"release": "2026-11-12-qwen3-8b-r5",
"model_commit": "<commit-sha>",
"engine": "vllm/vllm-openai:v0.33.0",
"nvidia_driver_min": "580",
"served_name": "site-assistant",
"approved_by": "<name>",
"previous": "2026-10-20-qwen3-8b-r4"
}
How do you keep the model name stable across versions?
Serve every release under one fixed name, so clients never know which release answered.
Both engines support this. In vLLM, --served-model-name sets the name used in the API, and without it the name is the --model argument, which is usually a path that changes every release. In llama-server, the model id is the path passed with -m unless you set --alias (server README).
# vLLM, inside serve.env
VLLM_ARGS="/srv/ai/current/model --served-model-name site-assistant --max-model-len 16384"
# llama.cpp
LLAMA_ARGS="-m /srv/ai/current/model/model-Q4_K_M.gguf --alias site-assistant -c 16384"
If the release name leaks into the client, every rollback becomes a client change too. At fifty sites, that is fifty changes during an incident.
How do you switch model versions atomically?
Point the service at a current link and replace that link with a rename.
On Linux, rename(2) replaces an existing path atomically, “so that there is no point at which another process attempting to access newpath will find it missing” (man page). Build the new link beside the old one, then rename it over the top:
switch_release() {
ln -s "releases/$1" /srv/ai/current.next
mv -T /srv/ai/current.next /srv/ai/current
systemctl restart ai-serve
}
switch_release 2026-11-12-qwen3-8b-r5 # update
switch_release 2026-10-20-qwen3-8b-r4 # roll back
The systemd unit reads /srv/ai/current/serve.env and starts the engine from /srv/ai/current. It never names a release. Update and rollback become the same operation in opposite directions, which means the rollback path gets tested every time you update.
An update you cannot roll back is a bet, not a release.
How do you canary a new model with no network?
Run the new release beside the old one, on a second port, before you switch anything.
- Start it. Launch the new release on another port with the same GPU budget. If the machine cannot hold both, canary on a second machine at the site or during a planned quiet window.
- Wait for health. vLLM’s
GET /healthreturns 200 when the engine is up and 503 when it has died. llama-server’s/healthreturns 503 with “Loading model” until the model is loaded, then 200. - Check the version. vLLM’s
GET /versionreturns the engine version. llama-server’sGET /propsreturnsbuild_info,model_pathand the chat template in use. Compare them withrelease.json. - Run the golden set. Send 20 to 50 real prompts from that site’s work, each with an expected answer or a check, to both releases. Score the results.
- Watch memory and latency. Send your longest real prompt at your real concurrency and confirm the server stays up.
Do not diff outputs word for word. vLLM’s reproducibility guide says it only provides reproducible results “when it runs on the same hardware and the same vLLM version”. A new release will change wording. Check whether the answer is right, not whether it is identical.
until curl -sf http://127.0.0.1:8001/health >/dev/null; do sleep 5; done
curl -s http://127.0.0.1:8001/version # vLLM
curl -s http://127.0.0.1:8081/props | jq '{build_info, model_path}' # llama.cpp
When should you roll back a model update?
When any trigger you agreed before the update fires. Decide them in advance, because nobody judges a threshold well during an incident.
- The new release does not reach a healthy
/healthwithin a set time, for example ten minutes. - It scores lower than the previous release on the golden set.
- It starts returning errors, timeouts or out of memory failures the old release did not.
- The engine or build reported by the server does not match the release record.
- An operator reports wrong answers on real work.
One trigger is enough. Roll back first and investigate on the old release, while the site keeps working.
How do you roll back, step by step?
The same way you updated, pointing the other way.
- Run
switch_releasewith thepreviousrelease fromrelease.json. - Wait for
/healthto return 200. - Confirm
/versionorbuild_infomatches the old release. - Rerun the golden set and check the scores match the old baseline.
- Write the rollback and its reason into the record.
- Keep the failed release on disk until someone has worked out why it failed.
On a single machine this is one restart, so the outage is however long your model takes to load.
What should the record of a model update contain?
Everything you would need to prove later which model answered, and why it changed.
| Field | Example |
|---|---|
| Release ID | 2026-11-12-qwen3-8b-r5 |
| File checksums | SHA256SUMS for the release folder |
| Model commit and engine version | <commit-sha>, vllm/vllm-openai:v0.33.0 |
| Driver branch | 580 |
| Approved by and when | Name, date and time |
| Switch time per machine | One line each time current changes |
| Canary result | Golden set score, old and new |
| Rollback | Time, trigger and who made the call |
Keep it on the site, in an append-only file or log, and copy it out when a link is available. A record that lives only at headquarters does not exist during the outage that needs it.
Next step
Rollback is one part of running AI at sites with no network. Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off, and it keeps a durable record of what ran. See how the platform works, or talk to our engineers about how you ship and roll back models today.