[ On the machine ]

Built into every machine.

Probes the GPUs, CPUs and accelerators, places the AI workload, tiers memory across VRAM and system RAM and exposes one API. Field server to rack.

Talk to the team How it works

s88 serve --model Llama-3.3-70B-AWQ

INITIALIZING

Llama-3.3-70B-AWQ

AWQ INT4 70B · TP=2

Detected

Backend Selection

Auto

llama.cpp vLLM Native

Memory Hierarchy

PASS

2x RTX 4090 VRAM (hot)

44.1 / 48 GB

System RAM (warm)

8.0 / 64 GB

Serving

localhost:8088/v1/chat/completions

Throughput

34.8 tok/s

Latency

29 ms

OOM Events

Uptime

[ Capabilities ]

Everything the machine needs.

Fig 2.1

Tiered memory orchestration

Weights and cache move between VRAM and system RAM. Workloads run on hardware that would not normally hold them.

Fig 2.2

Engine auto-selection

vLLM, llama.cpp and TensorRT-LLM run on Sector88, alongside vision, speech and sensor runtimes. You bring the workload. Sector88 places it on the right engine and silicon. When a faster engine lands, you inherit it.

Fig 2.3

OpenAI-compatible API

Drop-in for the OpenAI endpoint. Point existing software at a local URL. Chat, completions and tool calls.

Fig 2.4

Built for offline

Licences validate offline. Prompts and outputs never leave the box. Install over any medium, run on an empty network.

Fig 2.5

Hardware probing

Sector88 inspects the box on first start and configures itself. A CPU-only field server, a Jetson on a vehicle running ROS 2, or a rack of H100s. Same install.

Fig 2.6

Preflight validation

Sector88 validates the workload fits before loading. Catches OOM before it happens. No manual calculation. It checks the machine and tells you what works.

[ Comparison ]

Sector88 vs the DIY stack.

Engines run models. Sector88 runs the compute underneath them: probing the hardware, placing workloads, tiering memory and exposing one API on every node.

Feature

Sector88

Typical OSS

Memory orchestration

GPU → CPU memory, automatic

Single tier only

Engine support

llama.cpp, vLLM, TensorRT-LLM, native

One engine per tool

Hardware config

Auto-probes and adapts

Manual tuning required

API compatibility

OpenAI-compatible

Varies by engine

Offline operation

Built to run offline

Often requires cloud

One layer across the metal.

Every major GPU, CPU and edge accelerator. One install.

Hardware Platforms

NVIDIA CUDA

Validated

AMD ROCm

Supported

Intel Gaudi / Xeon

Supported

Google TPU

Supported

Qualcomm AI

Supported

Apple Silicon

Supported

CPU Servers

Supported

View all supported hardware →

Inference Backends

PyTorch Supported

Native inference

vLLM Supported

PagedAttention optimization

llama.cpp Supported

GGUF models, CPU/GPU

TensorRT-LLM Supported

NVIDIA optimization

vLLM on ROCm Supported

AMD Instinct, FP8/FP16

OpenVINO Supported

Intel Arc and Core Ultra NPU

llama.cpp on Metal Supported

Apple Silicon, unified memory

View all supported backends →

Put AI compute where the work is.

One install. GPU, CPU, TPU or mixed. We meet the box where it is.

Talk to the team