Building an Inference Engine for the Non-GPU Era

02 Sep, 2026

Why I built it

I wanted to actually understand how a modern LLM inference server works — not just use vLLM, but reimplement the hard parts myself: continuous batching (packing requests into a GPU step efficiently instead of waiting for a whole batch to finish) and paged KV cache memory (block-based memory management so sequences can share a common prompt prefix instead of duplicating it). I called it iron-batch. Rust, built from scratch, scheduler and memory allocator with no assumptions baked in about what hardware sits underneath.

Why Rust

The scheduler manages a lot happening at once — sequences being admitted, evicted, and preempted while memory blocks get allocated and freed underneath them. That's exactly the kind of code where a small mistake in shared state turns into a bug that doesn't show up in testing and only appears under real, concurrent load. Rust's ownership rules catch a meaningful category of these mistakes at compile time instead of in production. It doesn't catch logic bugs — I found plenty of those the hard way — but for infrastructure code, removing that category of risk upfront felt worth the tradeoff.

Doubts along the way

For a while, my scheduler was only ever tested against a fake backend that simulated token generation without running any real model. It passed every test. It also proved nothing about whether the design actually worked with a real model attached, and I had to sit with that gap before I trusted anything I'd built. Wiring in a real model surfaced two bugs immediately that the fake backend had never once caught — one where a tensor had the wrong data type, one where an empty default value broke the model's attention computation. Both were invisible until a real model was involved. That was a useful, uncomfortable reminder that a clean design and a correct implementation aren't the same thing.

Things I did differently from vLLM

Not better — different, on purpose, for a different goal. A few concrete choices:

• Compute is behind one interface, not baked into the scheduler. The scheduler calls a single method — advance this sequence by one token — and never touches CUDA, PyTorch, or any hardware-specific code directly. vLLM's execution engine (currently Model Runner V2) is deeply integrated with NVIDIA's stack, which is the right call for vLLM's audience. iron-batch keeps that boundary strict on purpose, so what's on the other side of it can change — a mock, an HTTP call to a model server, eventually a non-standard accelerator — without touching the scheduler or memory allocator at all.
• Memory safety is enforced by the compiler, not by discipline. Rust's ownership rules rule out a category of concurrency bugs (data races, use-after-free) at compile time. vLLM's core is Python plus CUDA/C++ kernels — fast, mature, but the safety guarantees around shared state come from careful engineering, not the language itself.
• Smaller surface area. iron-batch's scheduler and allocator are around a thousand lines. That's not a virtue by itself, but it means the whole thing is auditable by one person in an afternoon, which was useful for actually understanding every decision it makes.

None of this makes iron-batch faster — measured throughput is in BENCHMARKS.md, and at single-request speed the two are close, with vLLM's advantage showing up properly under real concurrent load, which I haven't tested yet. These differences are about what iron-batch is for: a hardware-agnostic scheduling core, not a faster vLLM.

What I ditched from the legacy approach

The biggest thing I deliberately left out: any hardcoded assumption about the underlying compute layer. No CUDA calls, no PyTorch dependency, nothing GPU-specific anywhere in the scheduler or the memory allocator. Everything hardware-specific lives entirely behind the one interface mentioned above.

If you want to look at any of this more technically, here's the code: github.com/Dev-X25874/iron-batch

Real numbers, and what they show

I benchmarked iron-batch against vLLM (running Model Runner V2, its current execution engine — vLLM removed PagedAttention as a named path and replaced it with MRv2, which pipelines CPU scheduling and GPU execution instead of running them sequentially) on the same model, Qwen2.5-3B, on the same GPU.

Within about 3% of each other at single-request speed. Getting here took two real fixes: batching multiple tokens per network call instead of making one HTTP round-trip per token, and matching precision and GPU tier between the two engines for a fair comparison (an earlier, unfair version of this test had iron-batch running 4-bit quantized against vLLM's float16 — that gap disappears once precision is matched).

What this result doesn't show: neither engine's actual specialty gets exercised at concurrency 1. vLLM's pipelined scheduling and iron-batch's own continuous batching both exist to serve many concurrent requests efficiently — a single-request test measures raw decode speed with KV cache reuse, nothing more. That's the next test to run, not something this result already answers.

Every command and raw output behind these numbers is in BENCHMARKS.md, including how to reproduce it yourself against your own Modal deployment.