# How to roll back a bad LLM update offline

> Roll back a bad LLM update at a site with no network: pin every part, keep one model name, switch atomically, canary first and record what ran.

- URL: https://www.sector88.co/blog/roll-back-llm-model-update-offline
- Author: Taskin Karlar
- Published: 2026-11-12
- Updated: 2026-11-12
- Category: Guides
- Tags: model-updates, rollback, vllm, llama-cpp, offline-ai, ai-operations

## Quick answer

To roll back a bad model update at a site with no network, keep the previous release on disk. Package each release as one folder with the weights, engine version, flags and checksums. Run the server through one link, keep the model name clients use fixed, test new releases first, and switch back with one restart.

On 2 November 2026 vLLM shipped v0.33.0. Among 927 commits, one line matters to anyone running it at a remote site: release wheels and images are now built for CUDA 13.0 only, and the CUDA 12.9 builds are no longer published ([release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3)). NVIDIA's [release notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html) say CUDA 13 applications run on drivers from the 580 branch or newer.

Upgrade a site that runs an older driver, without updating the driver too, and the new build can fail to start. At headquarters you pull the old image again. At a site with no network, there is no again.

That is the whole problem with model updates offline. **Rollback only works if the last good release is already on the machine.** This guide shows how to package releases so you can roll back an LLM update with one switch and one restart, using vLLM or llama.cpp.

## What is a model rollback?

**A model rollback is returning a serving machine to the last release that was known to work: the same weights, engine, flags and chat template, byte for byte.**

The word "model" hides most of the risk. A new set of weights is only one kind of update. A new engine version, a new quantization file or one changed flag can change answers, memory use or whether the server starts at all. If you can only roll back the weights, you cannot roll back most of what breaks.

## What actually changes in a model update?

At least six things, and each needs to be pinned and recorded.

| Part | What can change | How to pin it | Where to read it back |
|---|---|---|---|
| Weights | New fine tune or new base model | Model commit hash, file checksums | Release record, `sha256sum -c` |
| Quantization | Different GGUF file or precision | Exact file name and checksum | llama.cpp `GET /props` shows `model_path` |
| Tokenizer and chat template | Prompt format, stop tokens | Ship the files in the release folder | llama.cpp `GET /props` shows `chat_template` |
| Engine | vLLM or llama.cpp version | Image tag or wheel lock, never latest | vLLM `GET /version`, llama.cpp `build_info` |
| Driver and CUDA | Which builds can run | Driver branch in the release record | `nvidia-smi` |
| Flags | Context, memory share, parallel slots | One env file per release | The release folder itself |

The 10 November guide to [fixing CUDA out of memory in llama.cpp](/blog/llama-cpp-cuda-out-of-memory-fix) shows why flags belong in that list. A fitted set of flags is only right for one model, one build and one GPU. Pin it per release, not per machine.

## Why is rollback harder at a site with no network?

Because every recovery move you use in the cloud assumes a download.

1. **You cannot pull the old version.** Registries, package indexes and Hugging Face are out of reach. If the old image was deleted to make room, it is gone.
2. **Upstream may stop publishing it.** vLLM's CUDA 12.9 builds stopped with v0.33.0 ([pull request 62575](https://github.com/vllm-project/vllm/pull/62575)). An old version you never saved may not be available even when you do have a link.
3. **Nobody may be on site.** A fix that needs someone to plug in a laptop can take days.
4. **The update itself is expensive to move.** Weights run to tens of gigabytes and often travel on a drive. You do not want to repeat that trip to undo a mistake.

The rule follows: never delete the last known good release until the new one has proven itself at that site.

## How do you package a release you can roll back?

Make each release one self-contained folder, and never edit it after it ships.

```text
/srv/ai/releases/
  2026-10-20-qwen3-8b-r4/
    model/              # weights, tokenizer, chat template
    engine/             # vllm-openai image tar, or the llama-server binary
    serve.env           # every flag and environment variable
    golden.jsonl        # canary prompts with expected checks
    release.json        # IDs, versions, approvals
    SHA256SUMS          # checksum of every file above
  2026-11-12-qwen3-8b-r5/
    ...
/srv/ai/current -> releases/2026-10-20-qwen3-8b-r4
```

The [offline setup guide](/blog/run-vllm-llama-cpp-offline-air-gapped) covers staging weights, wheels and images with no internet. Three habits from it matter most for rollback:

- **Download weights with a pinned `--revision`.** When you use `hf download --local-dir`, Hugging Face writes a `.cache/huggingface/download/` folder of metadata next to the files ([download guide](https://huggingface.co/docs/huggingface_hub/guides/download)). The first line of each `.metadata` file is the commit hash. Copy it into `release.json` before you ship.
- **Save the engine with the model.** `docker save` the exact image tag, or copy the llama-server binary built for that release. The engine is part of the release.
- **Write the checksum list last.** Run `sha256sum` over the whole folder, and check it on arrival with `sha256sum -c SHA256SUMS`.

A minimal `release.json` looks like this:

```json
{
  "release": "2026-11-12-qwen3-8b-r5",
  "model_commit": "<commit-sha>",
  "engine": "vllm/vllm-openai:v0.33.0",
  "nvidia_driver_min": "580",
  "served_name": "site-assistant",
  "approved_by": "<name>",
  "previous": "2026-10-20-qwen3-8b-r4"
}
```

## How do you keep the model name stable across versions?

Serve every release under one fixed name, so clients never know which release answered.

Both engines support this. In vLLM, `--served-model-name` sets the name used in the API, and without it the name is the `--model` argument, which is usually a path that changes every release. In llama-server, the model `id` is the path passed with `-m` unless you set `--alias` ([server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)).

```bash
# vLLM, inside serve.env
VLLM_ARGS="/srv/ai/current/model --served-model-name site-assistant --max-model-len 16384"

# llama.cpp
LLAMA_ARGS="-m /srv/ai/current/model/model-Q4_K_M.gguf --alias site-assistant -c 16384"
```

If the release name leaks into the client, every rollback becomes a client change too. At fifty sites, that is fifty changes during an incident.

## How do you switch model versions atomically?

Point the service at a `current` link and replace that link with a rename.

On Linux, `rename(2)` replaces an existing path atomically, "so that there is no point at which another process attempting to access newpath will find it missing" ([man page](https://man7.org/linux/man-pages/man2/rename.2.html)). Build the new link beside the old one, then rename it over the top:

```bash
switch_release() {
  ln -s "releases/$1" /srv/ai/current.next
  mv -T /srv/ai/current.next /srv/ai/current
  systemctl restart ai-serve
}

switch_release 2026-11-12-qwen3-8b-r5   # update
switch_release 2026-10-20-qwen3-8b-r4   # roll back
```

The systemd unit reads `/srv/ai/current/serve.env` and starts the engine from `/srv/ai/current`. It never names a release. Update and rollback become the same operation in opposite directions, which means the rollback path gets tested every time you update.

An update you cannot roll back is a bet, not a release.

## How do you canary a new model with no network?

Run the new release beside the old one, on a second port, before you switch anything.

1. **Start it.** Launch the new release on another port with the same GPU budget. If the machine cannot hold both, canary on a second machine at the site or during a planned quiet window.
2. **Wait for health.** vLLM's `GET /health` returns 200 when the engine is up and 503 when it has died. llama-server's `/health` returns 503 with "Loading model" until the model is loaded, then 200.
3. **Check the version.** vLLM's `GET /version` returns the engine version. llama-server's `GET /props` returns `build_info`, `model_path` and the chat template in use. Compare them with `release.json`.
4. **Run the golden set.** Send 20 to 50 real prompts from that site's work, each with an expected answer or a check, to both releases. Score the results.
5. **Watch memory and latency.** Send your longest real prompt at your real concurrency and confirm the server stays up.

Do not diff outputs word for word. vLLM's [reproducibility guide](https://docs.vllm.ai/en/latest/usage/reproducibility.html) says it only provides reproducible results "when it runs on the same hardware and the same vLLM version". A new release will change wording. Check whether the answer is right, not whether it is identical.

```bash
until curl -sf http://127.0.0.1:8001/health >/dev/null; do sleep 5; done
curl -s http://127.0.0.1:8001/version          # vLLM
curl -s http://127.0.0.1:8081/props | jq '{build_info, model_path}'   # llama.cpp
```

## When should you roll back a model update?

When any trigger you agreed before the update fires. Decide them in advance, because nobody judges a threshold well during an incident.

- The new release does not reach a healthy `/health` within a set time, for example ten minutes.
- It scores lower than the previous release on the golden set.
- It starts returning errors, timeouts or out of memory failures the old release did not.
- The engine or build reported by the server does not match the release record.
- An operator reports wrong answers on real work.

One trigger is enough. Roll back first and investigate on the old release, while the site keeps working.

## How do you roll back, step by step?

The same way you updated, pointing the other way.

1. Run `switch_release` with the `previous` release from `release.json`.
2. Wait for `/health` to return 200.
3. Confirm `/version` or `build_info` matches the old release.
4. Rerun the golden set and check the scores match the old baseline.
5. Write the rollback and its reason into the record.
6. Keep the failed release on disk until someone has worked out why it failed.

On a single machine this is one restart, so the outage is however long your model takes to load.

## What should the record of a model update contain?

Everything you would need to prove later which model answered, and why it changed.

| Field | Example |
|---|---|
| Release ID | `2026-11-12-qwen3-8b-r5` |
| File checksums | `SHA256SUMS` for the release folder |
| Model commit and engine version | `<commit-sha>`, `vllm/vllm-openai:v0.33.0` |
| Driver branch | 580 |
| Approved by and when | Name, date and time |
| Switch time per machine | One line each time `current` changes |
| Canary result | Golden set score, old and new |
| Rollback | Time, trigger and who made the call |

Keep it on the site, in an append-only file or log, and copy it out when a link is available. A record that lives only at headquarters does not exist during the outage that needs it.

## Next step

Rollback is one part of running AI at sites with no network. Sector88 is one system to deploy, run and govern AI at every site, connected or completely cut off, and it keeps a durable record of what ran. See [how the platform works](/platform), or [talk to our engineers](/contact) about how you ship and roll back models today.

## Frequently asked questions

### How do you roll back an LLM model version with no internet?

Keep the previous release on the same machine before you update. If each release is a folder holding the weights, the engine version and the flags, rolling back means pointing the server at the old folder and restarting it. Nothing needs to be downloaded, so it works the same with or without a network.

### How many old model versions should a site keep?

At least the last release that was known to work, kept until the new one has run cleanly for an agreed period at that site. Disk space is the usual limit, because one set of weights can be tens of gigabytes. Keep the known good release before you keep anything else.

### Can I test a new model by comparing its answers word for word with the old one?

Not reliably. vLLM's own documentation says results are only reproducible on the same hardware and the same vLLM version, so a new engine or new weights will change wording even when the answer is right. Score each prompt against an expected answer or a check instead of an exact string.

### Why did my vLLM upgrade fail on a machine that worked before?

The engine changed underneath the model. From vLLM v0.33.0, released 2 November 2026, wheels and images are built for CUDA 13.0 only, and NVIDIA says CUDA 13 applications need a driver from the 580 branch or newer. A host on an older driver needs the previous image to keep serving.

### What should a model update record contain?

The release ID, the checksums of every file, the model commit, the engine and driver versions, the flags, who approved the change, when each machine switched, the canary result and any rollback with its reason. Keep it on the site, so it exists even when the link is down.

## Sources

- [vLLM v0.33.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.33.0rc3), vLLM on GitHub
- [Drop CUDA 12.9 from the release pipeline (pull request 62575)](https://github.com/vllm-project/vllm/pull/62575), vLLM on GitHub
- [CUDA Toolkit release notes](https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html), NVIDIA
- [Reproducibility](https://docs.vllm.ai/en/latest/usage/reproducibility.html), vLLM
- [llama.cpp HTTP server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), ggml-org
- [Download files (huggingface_hub guide)](https://huggingface.co/docs/huggingface_hub/guides/download), Hugging Face
- [rename(2), Linux manual page](https://man7.org/linux/man-pages/man2/rename.2.html), man7.org
