Self-hosting open-weight models on cleared hardware with vLLM and TurboQuant

The default AI deployment story is to call a hosted API. For most of the federal work we do, that story is a non-starter: data residency, accreditation boundaries, contractual restrictions, and the fact that nobody wants to argue with a Security Control Assessor about whether OpenAI's terms of service count as a system interconnect. The alternative is to run the model on hardware that lives inside the same accreditation boundary as the data. That's a stack story before it's a model story, and the stack has actually gotten good in 2026.

vLLM is the open-source LLM serving engine that started at UC Berkeley and is now a CNCF-style multi-vendor project: Meta, Mistral, Cohere, IBM, and Red Hat all run production traffic on it. The headline architectural contribution is PagedAttention, which applies virtual-memory paging to the attention KV cache. Instead of pre-allocating contiguous memory per request, which fragments badly under variable context lengths, PagedAttention allocates the cache in fixed-size blocks. The original paper reported, and current production stacks confirm, that this eliminates 60–80% of the memory waste that plagued naive serving.

The V1 engine, which became the default during the 2025 releases, fixed the next bottleneck: scheduling overhead at high concurrency. The previous architecture copied intermediate tensors between GPU and CPU during scheduling. V1 pins host memory and uses direct DMA transfers: zero-copy scheduling. It also disaggregates prefill and decode into independent scheduling domains. Prefill is compute-bound and benefits from large batches; decode is memory-bandwidth-bound and latency-sensitive. Treating them as separate workloads keeps a long prefill job from blocking the decode steps of in-flight requests, which is how the engine sustains low tail latency under mixed traffic.

Two practical properties make vLLM the right base for cleared deployments specifically. First, the HTTP surface is OpenAI-compatible. Every internal tool already written against the OpenAI SDK moves over with a base-URL change; call it an afternoon of migration, not a quarter. Second, the engine supports essentially every quantization format that matters: FP8, MXFP8, MXFP4, NVFP4, INT8, INT4, GPTQ, AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO. That breadth means whatever the model author shipped, vLLM can probably serve it.

Hardware is the constraint that forces every other decision. A 70B-parameter reasoning model in FP16 wants roughly 140 GB of memory before any context (H100-class silicon and a non-trivial budget). Most on-prem rack space approved for cleared compute is one or two generations older. INT4 weight quantization (GPTQ, AWQ) is mature enough that we treat it as a default, and it brings a 70B model into the 35–40 GB range (single A100 80GB territory, or a pair of A100 40GBs with tensor parallelism).

But weights are not the whole memory story. At long context, the KV cache dominates. A 70B model serving a 100K-token context can use more memory for KV cache than for weights. Conventional KV cache quantization methods exist, and they all want calibration data, which in a cleared environment is exactly the part you can't move across the accreditation boundary. That's the gap TurboQuant fills.

TurboQuant is a 2025–2026 result out of Google Research (Amir Zandieh, Majid Daliri, Majid Hadian, and Vahab Mirrokni). The paper, "TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate" (arXiv:2504.19874), was published at ICLR 2026. The core mechanism is elegant: apply a random orthogonal rotation to a vector before quantizing it. The rotation guarantees that the resulting coordinates are distributed approximately i.i.d. according to a known Beta distribution on the sphere, which means a single closed-form quantizer is near-optimal for every coordinate: no calibration, no model-specific tuning, no learned codebook.

Applied to the KV cache, TurboQuant reports 3-bit keys and 2-bit values without accuracy loss on the needle-in-the-haystack benchmark and the LongBench suite (question-answering, summarization, code completion), tested on Gemma and Mistral. The headline compression number is roughly 6× over FP16. The practical implication is that a 70B model serving a 100K-token context now fits on hardware that previously could only serve it at 16K.

Calibration-free matters in a cleared environment in a way that's hard to overstate. Most KV cache quantization methods require running calibration data through the model to learn a basis (the QuIP / QuIP# line) or to estimate per-channel scales. In a cleared deployment, that calibration data is often sensitive or regulated, and even when it isn't, the data-handling story is enough work that programs defer the optimization. TurboQuant is data-oblivious. The rotation is just a fixed random transform; nothing about the deployment needs to know what the model is going to be asked. That changes the deployment graph from "can we get calibration data approved" to "can we apply a rotation."

The deployment we're standardizing on combines both layers: weights quantized to INT4 with GPTQ or AWQ, KV cache quantized with TurboQuant. There's an open-source TurboQuant implementation with Triton kernels and a vLLM integration (the 0xSero/turboquant project on GitHub) that we've been tracking. On our reference hardware (A100 80GB, Llama 3.3 70B INT4 + TurboQuant KV), we measure 80–110 tokens per second per stream at 4K context, with continuous batching pushing aggregate throughput several times higher under realistic mixed load. For operator-pace question answering across long ATO packages or doctrine corpora, that's well past the threshold of useful.

Two operational pieces that are unglamorous and load-bearing. First, the model artifact lifecycle. A quantized model is a build artifact and belongs in the same configuration management baseline as the code that invokes it. We sign the quantized weights at build time, pin the SHA-256 hash in the deployment manifest, and refuse to load anything else at runtime. Same discipline we apply to compiled binaries. The KV cache quantization parameters (rotation seed, bit budgets, codebook) are part of the same manifest. A reviewer can reproduce the exact deployment from the manifest alone.

Second, vLLM is a fast-moving project (minor releases are roughly monthly), and the V1 engine is still maturing. We pin a known-good version per program, write a deprecation policy into the SDP, and only roll forward when an upstream fix justifies it. Treating the serving layer as a versioned dependency with its own change-control package is the cost of running a real model in a real accreditation boundary. Programs that don't do this end up debugging emergent regressions inside an SCA review window, which is exactly the wrong place to be debugging.

Where this falls down: workloads that genuinely need a frontier-tier model. The gap between an INT4-quantized 70B open-weight and the latest closed-API reasoning model is real, and on the hardest reasoning tasks (multi-step planning under partial observability, novel mathematical reasoning), it shows. Our pattern is to scope workflows so the hard reasoning step is the part that runs offline against the cleared model with extra inference-time compute budget (longer chains, more candidates, more critic passes), and to architect around the gap rather than pretend it isn't there. The frontier API is a moving target. The local stack is a stable platform. They're complementary, not interchangeable.

The other place to be careful: TurboQuant's published evaluations cover specific model families and specific tasks. For a new model or a new workload, our practice is to run an in-domain eval suite against both the quantized and unquantized models before the quantized version goes into production. Calibration-free does not mean evaluation-free. The eval becomes part of the configuration management package, same as any other test result on the program.

The summary: open-weight models, served by vLLM, with INT4 weights and TurboQuant KV cache, on accredited on-prem GPUs, signed and pinned through the same CM discipline we use for code. None of these pieces are exotic individually. The thing that's new in 2026 is that all the pieces are mature enough at the same time. For programs that needed to argue with an SCA about hosted APIs, that combination is the door.