What Is an Inference Engine? Serving Explained

An inference engine is the software that runs a model on hardware and serves requests to it. It is not the model, and it is not the act of running the model. If the weights are the engine block, the inference engine is everything else under the bonnet: how requests are queued, how memory for the attention cache is allocated, how many requests share a forward pass, and how tokens get back to you. vLLM, SGLang, TensorRT-LLM, llama.cpp and MLX are inference engines. Swap one for another with identical weights and identical hardware, and your throughput and cost per token change substantially.

Where it sits

Three layers are easy to collapse into one, so it is worth separating them.

When you call a commercial API, all three exist and you see none of them. When you self-host, you pick the engine yourself, and it becomes one of the two or three decisions that determine whether the project is affordable.

This is a different question from what inference is and why it costs money, which is about the economics of generating tokens at all. Here the question is narrower: given that you are generating tokens, what software does it, and what does that choice change.

What an inference engine actually does for you

A naive implementation loads the weights and processes one request at a time. That wastes most of a GPU, because generating a token is memory-bound rather than compute-bound: the hardware spends its time moving weights, not multiplying them. An inference engine exists to fill that idle capacity.

Four mechanisms do most of the work.

Batching requests together. Several requests ride the same forward pass. Since the pass is dominated by reading weights from memory, serving eight requests costs barely more than serving one. Continuous batching goes further and lets new requests join a batch already in flight rather than waiting for the slowest member to finish.

Managing the attention cache. Every token generated adds to a key-value cache that must stay in memory. Naive allocation reserves the maximum possible length per request and wastes most of it. vLLM's documentation describes PagedAttention as efficient management of attention key and value memory, which is the same idea as virtual memory paging: allocate in small blocks as needed instead of one contiguous worst-case reservation. This is usually the single biggest win, because it decides how many concurrent requests fit on a card.

Reusing shared prefixes. If a thousand requests begin with the same system prompt, prefix caching computes it once. The related prefill and decode split is what makes this possible.

Speculative decoding. A small draft model proposes several tokens, the real model verifies them in one pass, and accepted guesses come free. vLLM lists n-gram, suffix, EAGLE and DFlash methods. When Meta launched Muse Spark Open, the third-party speed figures on its launch page were speculative decoding results rather than model-quality results, which tells you how much of serving performance now lives at this layer.

Why the same model behaves differently on different engines

Identical weights plus a different engine gives you different latency, different throughput at a given concurrency, different memory headroom, and sometimes different output.

That last one surprises people. Engines apply quantization, and vLLM alone lists FP8, INT8, INT4 and GPTQ or AWQ formats. A model served at INT4 is arithmetically not the same model as one served at BF16. Add differences in default sampling parameters, in how batching affects floating-point reduction order, and in tokenizer handling, and you get the effect documented in why the same AI model gives different results on different providers. The engine is one of the causes.

Which engine for which job

Broad strokes, and the right answer depends on your hardware and your traffic shape rather than on a ranking.

• vLLM. The usual default for serving an open-weight model on GPUs. Broad model support, strong concurrency, large community.

• SGLang. Also aimed at high-throughput GPU serving, with particular attention to structured output and complex multi-call programs.

• TensorRT-LLM. NVIDIA's own, compiled per model and per GPU. The most work to set up and the best peak numbers on NVIDIA hardware.

• llama.cpp. CPU and mixed CPU-GPU, quantized GGUF builds, runs on hardware the others will not touch. The route to a model on a laptop.

• MLX. Apple silicon, using unified memory, which is why Mac Studio clusters appear in the specs of very large open-weight models.

Meta's Muse Spark Open launch guide names SGLang, vLLM, TensorRT-LLM, MLX, llama.cpp and Unsloth as supported serving paths, which is a fair snapshot of the current field.

When this matters to you, and when it does not

It does not matter if you call a hosted API and are happy with the price. The provider picked the engine. Your lever is the model and the prompt, not the serving stack.

It starts to matter the moment you consider running a model yourself, because the engine choice sets how much hardware you need for a given request rate, which is the dominant term in whether self-hosting beats an API at your volume. A model that needs four GPUs on one engine and six on another is a 50 percent cost difference from a software decision.

It also matters when you are reading a benchmark. A throughput number without the engine, the engine version, the quantization and the concurrency named is close to meaningless. Those four values move results more than most model-to-model gaps.

Frequently asked questions

Is an inference engine the same thing as an inference server?

In practice the terms are used interchangeably. If anyone distinguishes them, the engine is the part that executes the model and the server is the HTTP layer and scheduler around it. Most projects ship both together.

Does the inference engine change the model's answers?

It can. Quantization format, default sampling settings and batching-related numerical differences all shift output. The weights are the same; the computation is not exactly the same.

Do I need to know this if I only use an API?

No, with one exception: when comparing providers serving the same open-weight model, their engine and quantization choices explain most of the quality and speed differences you notice.

What is the quickest way to try one?

vLLM on a rented GPU for a server-shaped workload, or llama.cpp on the machine in front of you for a small quantized model. Both are an afternoon. Begin with what local AI models are good for if you are deciding whether to bother at all, and how AI models work for the layer below.