Fig. 5.0 · Performance
Throughput you can
stand behind.
Engine numbers at the top of the envelope. Field proof from our own runs under sustained load.
[ Field proof ]
Proven under load.
A multi-day run on 8x NVIDIA H100s under sustained traffic. It held up through an NVIDIA maintenance window. The AI kept running.
Measured by Sector88
55+ hrs
Continuous serving on 8x H100
Measured by Sector88
830K+
Completions served
Measured by Sector88
0
Failures. Zero restarts.
Measured by Sector88
70B
On a single 24 GB card. Zero OOM.
Sector88 evaluation runs, September 2026 (8x H100 NVL) and January 2026 (RTX 4090, Llama 3 70B, memory tiered across VRAM and system RAM).
[ Engine envelope ]
The engines on Sector88, at full throttle.
vLLM, TensorRT-LLM and llama.cpp run on Sector88, and it picks the right one for each machine. These are the published numbers from the engine and hardware makers, not Sector88 measurements. Your numbers come from your box.
TensorRT-LLM
Published
27,028
tok/s
1x NVIDIA H200
Llama 3.1 8B FP8. Output tokens per second at full load.
Source: NVIDIA TensorRT-LLM performance overview ↗vLLM on ROCm
Published
23,515
tok/s
8x AMD Instinct MI300X
Llama 2 70B, offline. AMD MLPerf Inference v4.1 submission.
Source: AMD ROCm blog, MLPerf Inference v4.1 ↗TensorRT-LLM
Published
10,614
tok/s
1x NVIDIA B200
Llama 3.3 70B FP4 on a single GPU. Output tokens per second at full load.
Source: NVIDIA TensorRT-LLM performance overview ↗TensorRT-LLM
Published
7,178
tok/s
4x NVIDIA B200
Llama 3.1 405B FP4. 128 in, 2,048 out.
Source: NVIDIA TensorRT-LLM performance overview ↗vLLM
Published
6,067
tok/s
1x NVIDIA H100 80GB
Llama 3.1 8B. Aggregate across concurrent chat requests.
Source: Microsoft Azure HPC benchmarks ↗llama.cpp
Published
179
tok/s
Apple M5 Ultra
Llama 2 7B Q4_0, single stream. 4,945 tok/s prompt processing.
Source: llama.cpp Apple Silicon benchmarks ↗Engine and hardware vendor figures as published. Throughput depends on model, precision, batch and sequence length. MLPerf is a trademark of MLCommons Association.
[ Datacentre ]
Throughput at scale.
On datacentre GPUs, Sector88 drives vLLM with continuous batching and memory paging. These are the engine's published numbers. Run it on your box to see yours.
vLLM engine
High-throughput continuous batching. Default for datacentre and modern NVIDIA GPUs.
Llama 3.1 8B
| Hardware | Workload | Generation tok/s |
|---|---|---|
| H100 80GB | Chat, concurrent | 6,067 |
| A100 80GB | Chat, concurrent | 2,622 |
| A100 40GB | Chat, concurrent | 2,459 |
| L4 24GB (FP8) | Chat, concurrent | 1,800 |
Source: Microsoft Azure HPC benchmarks, Llama 3.1 8B (2025). Aggregate throughput across concurrent requests.
TensorRT-LLM engine
NVIDIA's optimised runtime. Selected on Hopper and Blackwell when it is the fastest path for the workload.
Llama 3.3 70B FP8
| Hardware | Sequence length | Generation tok/s |
|---|---|---|
| 2x H200 | 128 in, 2048 out | 7,467 |
| 2x H200 | 128 in, 128 out | 6,328 |
| 2x H100 | 128 in, 128 out | 6,092 |
| 2x H200 | 2048 in, 2048 out | 3,776 |
| 2x H100 | 2048 in, 2048 out | 2,786 |
Source: NVIDIA TensorRT-LLM performance overview (current release). FP8, tensor parallelism = 2.
[ Memory orchestration ]
Serve models larger than your GPU.
Traditional KV-cache management wastes 60 to 80% of GPU memory through fragmentation. Sector88 pages that memory and tiers it across VRAM and system RAM, so you can serve models that exceed your VRAM without crashing.
Throughput vs naive serving
24x
Published PagedAttention result versus HuggingFace Transformers (Kwon et al.).
Memory recovered
60-80%
Published PagedAttention result: memory older serving systems lost to KV-cache fragmentation.
Memory hierarchy
VRAM + RAM
VRAM for hot pages, system RAM for the overflow. Graceful spill, no manual configuration.
Based on PagedAttention (Kwon et al., SOSP 2023, arXiv:2309.06180). Sector88 builds on it and tiers across GPU and CPU memory.
[ Hardware ]
The right engine for every tier.
Sector88 probes the machine on first start, selects the optimal engine, and configures memory tiering automatically. You do not choose the engine. You get the fastest one for your box.
| Tier | Hardware | Engine selected | Model capacity |
|---|---|---|---|
| Datacentre | H100, H200, A100 80GB, MI300X | vLLM or TensorRT-LLM | 70B+ full precision, 405B with tensor parallelism |
| Inference card | L4, L40S, A10, RTX 4090 | vLLM | Up to 70B quantized with paging, 8-20B full precision |
| Workstation | RTX 30/40 series, single-card | vLLM or llama.cpp | 8-13B comfortably, 70B quantized with tiering |
| Edge | Jetson Orin AGX, NX, Nano | llama.cpp | 1B to 8B quantized, multimodal where supported |
| CPU-only | x86 Xeon, EPYC, ARM | llama.cpp | 3-8B quantized at interactive speed |
[ What we add ]
Built to stay up.
Throughput is the floor. These are the things that keep inference stable under real-world load.
Automatic engine selection
Probes hardware on boot. Picks the fastest engine for your card. No configuration required.
Memory tiering
Pages KV-cache across VRAM and system RAM. Models that exceed your VRAM still serve without crashing.
OOM protection
Refuses configurations that would crash. Degrades gracefully under load instead of falling over.
OpenAI-compatible API
One endpoint regardless of engine or hardware. Your application code never changes.
[ Models ]
Supported models.
If it runs in vLLM, llama.cpp or TensorRT-LLM, it runs in Sector88. Bring your own fine-tuned weights. We handle the rest.
Llama 3.1 / 3.2 / 3.3
1B to 405B. The most widely deployed open model family.
Mixtral 8x22B / 8x7B
Mixture-of-experts. High quality at efficient compute cost.
Qwen 2.5
Strong on coding and mathematics. Multiple size variants.
Phi-4
14B. Efficient for structured tasks on smaller hardware.
DeepSeek V3 / R1
Reasoning-focused. Permissive licence.
Gemma 2 / 3
Google. Efficient open-weights with broad support.
Custom fine-tunes
GGUF, SafeTensors, HuggingFace format. Bring your own weights.
Quantised or full precision
GGUF Q4 to Q8, FP8 and FP16. Sized to fit the box.
Something else?
If the engine supports it, we run it. Talk to us about your specific model.
Benchmark it on your workload.
Tell us your site, your model, and your workload. We will run Sector88 in your environment and show you exactly what it delivers.