AI where the cloud can't reach.

1 · The problem we built for

Most LLM serving stacks assume an environment Sector88 customers do not have: continuous internet, homogeneous GPUs, an SRE team on standby, and an appetite for paying per token. Our customers run AI at ground stations, air-gapped facilities, remote industrial sites, and SCIFs. The cloud is not an option. Their hardware is what is on the truck.

The open-source serving ecosystem is excellent. vLLM, llama.cpp, and TensorRT-LLM all ship state-of-the-art performance. But they are not operations products. They do not probe hardware on boot, page memory across tiers when VRAM runs short, select the right engine for the silicon underneath, or roll back cleanly when an update fails at 2am in a disconnected facility.

Customers running on real hardware in real environments need probing, selection, paging, monitoring, fleet control, signed bundles, rollback, and someone to call when a Jetson thermals out. They need a single API in front of the model that fits their box, not a different shape per engine. They need the platform layer that turns those engines into infrastructure.

Sector88 is that layer.

2 · Architecture overview

The platform has two long-running components and one engagement model.

2.1 Runtime

Runtime is the per-node service. One Runtime per inference box. On first start it probes the hardware: GPU vendor, VRAM capacity, driver version, CPU instruction set extensions, available RAM, NVMe bandwidth. Based on that probe it selects an engine, loads the model into the memory tier that fits, and serves an OpenAI-compatible API.

Runtime is the only component that touches the GPU. It manages the full lifecycle of inference on that node: model loading, memory allocation across tiers, request batching, health monitoring, thermal throttle response, and graceful degradation under load.

It ships as a single container image (or bare-metal binary for air-gapped installs). There is no JVM, no Python runtime in the critical path, and no dependency on external package managers at install time.

2.2 Hub

Hub is the control plane. It manages the fleet of Runtimes: deployment, configuration, model distribution, secrets, telemetry aggregation, and alerting. Hub never opens an inbound port on a Runtime. Communication is always outbound from the node. Loss of connectivity between Runtime and Hub is a connectivity event, not a security event and not a service interruption. Runtimes serve indefinitely without Hub.

Hub provides the operator dashboard: fleet topology, per-node health, model versions, request throughput, memory utilisation, and alert history. It is where operators define fleet-wide policies (which model goes where, which nodes get priority updates, what the thermal ceiling is).

2.3 Forward-deployed engineering

The install motion is not self-serve by default. A senior Sector88 engineer comes onto the engagement, runs pre-flight against the customer's hardware, stands the platform up against their workloads, hardens it for their environment, validates performance against target SLAs, and hands off with documentation and runbook access. This is not indefinite: the engagement has a defined scope, a defined handover, and the customer operates independently afterwards with support tiers attached.

2.4 Component interaction

Runtime and Hub communicate over a single outbound mTLS connection initiated by the node. The protocol is pull-based: Runtime checks in on a configurable cadence, receives any pending configuration changes, and pushes telemetry. Air-gapped deployments run Runtime standalone, or with a local Hub instance on the same isolated network. In either case, inference never depends on control-plane availability.

3 · Memory orchestration

The single most common failure mode for self-hosted inference is the box running out of memory. A 70B parameter model at FP16 requires approximately 140GB of working memory. Most edge GPUs ship with 24GB or less. The standard industry response is aggressive quantization: compress the model to INT4 and accept the quality degradation. We take a different approach.

3.1 Tiered memory model

Sector88 Runtime treats memory as a hierarchy. VRAM is tier zero: fastest, smallest, most expensive. Host RAM is tier one: 10x larger than VRAM on most boxes, two orders of magnitude slower for GPU-bound operations, but fast enough for pre-staging. NVMe is tier two: effectively unbounded, limited by sequential read bandwidth (typically 5-7 GB/s on PCIe 4.0 drives).

The orchestrator places model layers, KV-cache pages, and intermediate activations across these tiers based on access pattern, latency budget, and available capacity. Hot layers stay in VRAM. Warm layers sit in RAM, pre-fetched into VRAM before they are needed. Cold layers live on NVMe and are streamed on demand.

3.2 Layer-aware scheduling

Traditional inference engines load the entire model into GPU memory and fail if it does not fit. Runtime splits the model into its transformer layers and schedules them through the GPU sequentially. The next layer is prefetched from RAM or NVMe while the current layer is computing. This is not random paging. It is deterministic scheduling based on the model's known forward-pass graph. The sequence of layer accesses during inference is perfectly predictable, which means prefetch is always correct.

3.3 KV-cache management

KV-cache fragmentation alone wastes 60 to 80 percent of GPU memory on traditional servers (Kwon et al., 2023). vLLM solved this on the GPU with PagedAttention, recovering the slack and pushing throughput up by an order of magnitude versus naive serving. Sector88 extends the same principle past the GPU boundary.

For long-context inference, the KV cache grows linearly with sequence length and can exceed VRAM capacity before model weights do. Runtime offloads the oldest KV-cache pages to host RAM using a ring buffer. Eviction priority is based on attention entropy: low-entropy heads (those whose attention patterns have stabilised) are evicted first, preserving the most information-dense state in fast memory.

The practical result: a 70B model on a single L4 (24GB VRAM) is a real, serving configuration. Our reference deployment with AI Sweden runs gpt-oss-20b on a single NVIDIA L4 at 43 tok/s sustained.

3.4 OOM protection

Runtime monitors memory pressure across all tiers continuously. When VRAM utilisation crosses a configurable threshold (default 92%), the orchestrator begins proactive eviction of cold pages before the allocator hits a hard wall. If a request would push the system past capacity, it is queued rather than crashed. The operator sees a latency spike, not a dead node.

In extreme cases (thermal throttling reducing effective compute, sudden memory pressure from OS-level processes), Runtime enters a graceful degradation mode: it reduces batch size, increases inter-token latency, and alerts Hub. It does not crash. It does not lose in-flight requests.

4 · Engine selection

We ship three inference engines today. The platform routes to the correct one automatically based on hardware probe results.

4.1 vLLM

Default engine for most NVIDIA datacentre and inference cards: H100, H200, A100, L4, L40S, RTX 30/40 series. Also runs on AMD MI300X via ROCm and Intel Gaudi via the HPU plugin. vLLM is the right choice for high-throughput batched serving where continuous batching and PagedAttention deliver maximum tokens per second per dollar.

Reference numbers: ~6,067 tok/s on H100 for Llama 3.1 8B (FP16, batch 64). ~2,622 tok/s on A100 80GB for the same model. ~1,800 tok/s on L4 with FP8 quantisation.

4.2 TensorRT-LLM

Used on Hopper and Blackwell architectures where the customer wants peak throughput and is willing to commit to NVIDIA-specific model compilation. TensorRT-LLM fuses operations, applies kernel-level optimisations, and exploits hardware-specific features (FP8 on Hopper, FP4 on Blackwell) that are not available to generic engines. We deploy it on H100/H200 clusters and on Jetson Orin AGX where TRT-LLM outperforms llama.cpp on larger models.

Reference numbers: ~7,467 tok/s on 2x H200 for Llama 3.3 70B FP8 (long-output regime, tensor parallelism across 2 GPUs).

4.3 llama.cpp

Runs everywhere else: Jetson Nano, Jetson Orin NX, Apple Silicon, ARM servers, x86 CPU-only sites, AMD consumer cards. Quantised GGUF models on constrained hardware. llama.cpp is the only engine that delivers usable inference on devices with no discrete GPU. On Jetson Orin NX, Llama 3.2 1B runs at 17 to 20 tok/s. On Apple M3 Max, Llama 3.1 8B Q4_K_M runs at ~45 tok/s. On AMX-equipped Xeon CPUs, it saturates the memory bandwidth for interactive use.

4.4 Automatic selection

The customer never picks an engine. Runtime probes the box on first start, identifies the best-fit engine based on silicon vendor, driver version, available VRAM, and model requirements, and serves. When a faster engine ships next quarter, the next update picks it up automatically. The customer's API contract does not change. Their application code does not change. Their integration does not break.

5 · API layer

Runtime exposes an OpenAI-compatible API on every node. Any application, SDK, or framework that speaks to the OpenAI API speaks to Sector88 without modification. This includes LangChain, LlamaIndex, Semantic Kernel, the OpenAI Python and Node SDKs, and any custom HTTP client.

Supported endpoints: /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models. Streaming via SSE. Function calling. JSON mode. All parameters that the underlying engine supports are passed through transparently.

The API layer handles request routing, load balancing across multiple models on the same node (if memory allows), authentication via API keys or mTLS client certificates, rate limiting, and request queuing. It is the single integration surface for all downstream applications regardless of which engine is running beneath it.

6 · Fleet operations

Hub manages a fleet of Runtimes from one place. A small fleet might be three nodes in one server room. A large one might be 200 GPUs across three continents and four classification levels. The control plane is the same.

6.1 Declarative configuration

Every node's desired state (model, engine override if any, capacity limits, tier policy, thermal ceiling) is described in configuration and reconciled continuously. Operators define intent. Hub converges the fleet toward that intent. Drift is detected and corrected automatically.

6.2 Outbound-only telemetry

Nodes always initiate. Hub never reaches into a Runtime. The node opens a single outbound mTLS connection and pushes telemetry on a configurable cadence. If the network path is severed, telemetry queues locally and replays when connectivity resumes. There is no requirement for Hub to be reachable for inference to continue.

6.3 Topology and grouping

Operators organise nodes into groups by site, classification level, hardware class, workload type, or any other taxonomy. Policies apply at the group level. A "defence-edge" group might have aggressive thermal limits and zero-egress enforced. A "datacentre-batch" group might prioritise throughput over latency. The same Hub manages both.

6.4 Secrets and identity

SSO via SAML or OIDC. Role-based access control mapped to the customer's identity provider groups. Hub does not own or store user credentials. API keys for Runtime access are generated per-node or per-group, rotated on schedule, and revocable from Hub. mTLS client certificates are supported for machine-to-machine authentication in zero-trust environments.

6.5 Disconnected operation

Runtimes serve indefinitely without Hub. The node has everything it needs locally: model weights, engine binary, configuration, API keys. Hub is for management, not for serving. Telemetry replays on reconnect. Configuration changes queue and apply on next check-in. This is the fundamental architecture decision that makes the platform viable for intermittent and air-gapped environments.

7 · Deployment patterns

Four reference patterns, in increasing isolation. The same Runtime binary runs in all four. Only the network posture and update cadence change.

7.1 Cloud-connected

Outbound TLS to Sector88-hosted Hub. Fastest time to production. Customer provides hardware (cloud VMs, bare metal, or on-prem), we provide the platform. Telemetry flows continuously. Updates deploy within hours of release. Appropriate for teams that want managed operations without sending inference traffic to a third party.

7.2 On-prem connected

Hub self-hosted on the customer's infrastructure (VPC, bare metal, private cloud). Outbound-only TLS from each Runtime to that Hub. No data leaves the customer's network. Default for regulated industries: financial services, healthcare, government. The customer operates Hub themselves or delegates to a managed services partner.

7.3 Edge / intermittent

Connectivity is available but unreliable. Runtimes serve continuously regardless of link state. Telemetry queues locally (configurable retention, default 7 days) and replays when the link is up. Configuration changes apply on next successful check-in. Typical environments: ground stations with periodic satellite uplink, remote well-pads with cellular, branch offices with unreliable WAN.

7.4 Air-gapped

No external network at all. Zero bytes cross the boundary except on operator-approved cross-domain media. The install process: signed release bundle is built in a connected environment, transferred to the air-gapped facility via approved media (USB, optical, data diode), verified via offline signature check (GPG, cosign), installed, and validated. Updates follow the same path. Local Hub, local container registry, local model storage. The entire platform operates indefinitely with no external dependency.

Default deployment pattern for defence, classified environments, critical national infrastructure (CNI), operational technology (OT) networks, and any facility where network isolation is a security requirement rather than a limitation.

8 · Updates and rollback

Platform and model updates roll across the fleet in staged waves. The operator defines the rollout policy: percentage of nodes per wave, health-check gate between waves, automatic rollback threshold. A failed health check on any node in a wave halts the rollout and triggers automatic rollback of that wave to the previous known-good state.

Rollback is atomic per node. The previous Runtime version and model weights are retained locally until the new version passes health checks. If the new version fails (crash, OOM, degraded latency beyond threshold), the node reverts in under 30 seconds without operator intervention. Hub records the event and alerts.

For air-gapped environments, the same staged rollout applies within the isolated network. The operator triggers the rollout from local Hub. The process is identical; only the source of the update bundle differs.

9 · Security posture

The platform is designed for environments where security is not a feature but a prerequisite for deployment. The full security breakdown lives at /security. The architectural highlights:

9.1 Data sovereignty

Inference content (prompts, completions, embeddings) never leaves the node. There is no mechanism in the platform to exfiltrate inference data. The audit log is metadata-only by design: timestamps, token counts, latency, model name. Content is never logged, never stored beyond the request lifecycle, never transmitted.

9.2 Zero egress

On paid tiers, zero telemetry leaves the node by default. The operator explicitly opts into outbound telemetry and controls exactly what is transmitted (metadata only, aggregated metrics, or nothing). Air-gapped deployments have zero network egress by physical constraint.

9.3 Supply chain integrity

Every release ships with a Software Bill of Materials (CycloneDX and SPDX formats). Build provenance is SLSA Level 2, signed with cosign. Container images are signed and verified at pull time. For air-gapped installs, the release bundle includes all dependencies and their verified hashes. There is no runtime fetch from external repositories.

9.4 Compliance alignment

SOC 2 Type II and ISO 27001 certification in progress. Additional alignment work in flight: CMMC Level 2 (for US DoD supply chain), IRAP at PROTECTED level (for Australian government), CSA STAR Level 1, and CISA Secure by Design principles. Source-available access and code escrow arrangements are available for mission customers on negotiated terms.

9.5 Network security

All control-plane communication is mTLS with certificate pinning. Runtime never accepts inbound connections from Hub or any external source (unless the operator explicitly configures the API to listen on a network interface). API access is authenticated via API keys, mTLS client certificates, or both. Rate limiting and request size limits are configurable per endpoint.

10 · Observability

Every Runtime exposes Prometheus-compatible metrics on a local scrape endpoint. Key metrics:

• Throughput: tokens per second (generation), requests per second, batch utilisation.
• Latency: time to first token (TTFT), inter-token latency (ITL), end-to-end request latency, queue wait time.
• Memory: VRAM utilisation, RAM utilisation, NVMe tier usage, KV-cache occupancy, page eviction rate.
• Hardware: GPU temperature, GPU power draw, GPU clock speed, thermal throttle events, fan RPM (where available).
• Availability: uptime, health check pass/fail, last successful inference timestamp, error rate by type.

Hub aggregates these metrics across the fleet and exposes them via dashboard, Prometheus federation, or webhook-based alerting. Operators can integrate with existing monitoring stacks (Grafana, Datadog, Splunk, or any Prometheus-compatible consumer) without replacing their tooling.

Structured logs (JSON, configurable verbosity) are written locally and optionally forwarded to Hub or a customer-operated log aggregator. Log content is metadata-only. Inference content is never present in logs regardless of verbosity level.

11 · Performance

Throughput is dominated by hardware, model size, quantisation level, and batch size. Sector88 does not add meaningful overhead above the underlying engine. The orchestration layer (hardware probe, memory tiering, API routing) adds less than 2ms to end-to-end request latency in steady state.

11.1 Reference benchmarks

These are real numbers from real deployments, verified by customers or independently published sources. Full benchmark methodology and expanded results at /platform/benchmarks.

• 43 tok/s on a single NVIDIA L4 (24GB), gpt-oss-20b, measured by AI Sweden.
• ~6,067 tok/s on H100 (80GB), Llama 3.1 8B FP16, vLLM engine, batch 64.
• ~7,467 tok/s on 2x H200 (141GB each), Llama 3.3 70B FP8, TensorRT-LLM, tensor parallel.
• ~2,622 tok/s on A100 80GB, Llama 3.1 8B FP16, vLLM engine.
• 17-20 tok/s on Jetson Orin NX (16GB), Llama 3.2 1B Q4, llama.cpp engine.

11.2 The calculator

The sizing calculator at /calculator turns these reference numbers into a sizing answer for any workload: model class, target latency, concurrent users, hardware envelope. It is the tool we use internally during pre-sales conversations and it is available publicly.

11.3 What we add beyond raw engine speed

Raw tok/s is not what makes a deployment production-ready. What Sector88 adds above the engine: sustained throughput under variable load (continuous batching tuned per-hardware), zero-crash memory management (OOM protection, graceful degradation), thermal-aware scheduling (clock speed backing off before thermal shutdown), automatic engine selection (always the fastest available for the hardware), and fleet-wide consistency (every node performs to its hardware ceiling, not to the lowest common denominator).

12 · Commercial model

12.1 Deployment models

Three options: self-hosted (customer operates, we support), managed (we operate on customer hardware), and forward-deployed (we install, harden, hand off). Most enterprise customers start with forward-deployed and transition to self-hosted after the first production milestone.

12.2 Support tiers

Standard (business-hours email, 24h response SLA), Production (24/7 with 4h response, dedicated Slack channel, quarterly reviews), and Mission (24/7 with 1h response, named engineer, on-call escalation, annual security review). Mission tier includes source-available access and code escrow.

12.3 Pricing

Per Runtime node, annual term. No per-token fees, no inference metering, no usage-based surprises. The customer knows their cost at contract signature. Forward-deployed engineering is scoped and priced separately based on engagement complexity. Volume discounts apply at fleet scale.

13 · References

1. Kwon, Li, Zhuang, Sheng, Zheng, Yu, Gonzalez, Zhang, Stoica. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. arXiv:2309.06180.
2. Microsoft Azure HPC Community. Inference performance of Llama 3.1 8B using vLLM across various GPUs and CPUs. 2025. Microsoft Tech Community.
3. NVIDIA. TensorRT-LLM Performance Overview. nvidia.github.io/TensorRT-LLM.
4. NVIDIA Jetson AI Lab. Benchmarks for LLMs on Jetson Orin. jetson-ai-lab.com.
5. vLLM project. Documentation and supported hardware matrix. docs.vllm.ai.
6. Cloud Security Alliance. STAR Level 1 self-assessment program. cloudsecurityalliance.org/star.
7. NIST. SP 800-171 rev. 3: Protecting Controlled Unclassified Information. csrc.nist.gov.
8. AI Sweden. GPT-SW3: an open Swedish language model. ai.se/en/project/gpt-sw3.
9. CISA. Secure by Design: Shifting the Balance of Cybersecurity Risk. 2023. cisa.gov/securebydesign.
10. SLSA. Supply-chain Levels for Software Artifacts, v1.0. slsa.dev.

© Sector88. May 2026. v1.2. This document is the canonical source. Version is incremented on substantive changes.