Benchmarks

QEngine, measured October 13–14, 2026

QEngine is my engine for running language models on one NVIDIA GH200 GPU. With one request at a time it was the fastest of five engines on the main test, but it is slower on very long prompts and with many requests at once. Like the others, it uses speculative decoding: a quick draft guesses several tokens (word pieces) ahead, and the full model checks them in one pass.

Results

Tokens per second, one request at a time: 325
Relative to mistral.rs, the fastest other engine at one request: 1.49×
Relative to vLLM with 32 requests at once (slower): 0.58×

The model is Qwen3.8-27B-FP8, an open 27-billion-parameter model. The main test uses the 64 writing prompts of mistral.rs’s published benchmark. QEngine was tuned on them, which can flatter it, so I also ran code, math and JSON prompts it was never tuned on: there it was 1.49× to 1.65× as fast as mistral.rs.

Tokens per second, one request at a time
Engine	Tokens per second	Speedup
QEngineOctober 13 build	325	Not applicable
mistral.rscommit b6ea3f2	214	1.49×1.46–1.53
vLLMv0.31	185	1.75×1.71–1.79
SGLangv0.5.19.dev99	177	1.82×1.79–1.85
TensorRT-LLMv1.3.0rc29	157	2.07×2.02–2.11

Medians of three runs, with 95% intervals. The mistral.rs and SGLang speedups use their faster earlier speeds (Method).

Relative to mistral.rs, one request at a time

Grey line: equal speed. Hollow: slower. Error bars: 95% intervals (prompts redrawn at random).

The speedup does not change the answers: every output checked matched a run without speculative decoding.

Writing: 64/64
Code: 24/24
Math: 24/24
JSON: 24/24
~4,000 tokens: 16/16
~16,000 tokens: 16/16
~30,000 tokens: 16/16

Writing: all 3 runs, compared with a run from an earlier QEngine build.

Where it is slower

Long prompts

From about 16,000 prompt tokens mistral.rs is faster, because QEngine is slower to read the prompt. Other engines were not run on long prompts. QEngine’s settings were tuned on these prompts, but reading long prompts got less optimization work than the rest of the engine before I ran out of GPU time; I plan to improve it and will update these numbers.

~16,000-token prompts, lower is better

Medians over 16 prompts, one run each.

Many requests at once

Here I compare total output across all requests with vLLM’s best setup for each number of requests.

Total tokens per second by requests at once, with QEngine relative to vLLM below

Requests at once double at each step. Single runs on 32 prompts from QEngine’s tuning pool, so no intervals.

Another way to compare: set a minimum average speed that 9 in 10 users must get, and see how much each engine can serve in total.

Most total tokens per second under a speed floor
Engine	10 ms per tokenabout 100 tokens/s per user	20 ms per tokenabout 50 tokens/s per user
QEngine	925	1,706
vLLM	1,816	2,933
TensorRT-LLM	859	2,058

The floor: 9 in 10 requests average at most that time per later token. TensorRT-LLM kept its one-request setup.

Not measured yet: memory use, startup time, cost per token.

Other models

On two smaller models, with the same 64 writing prompts and one request at a time, I compared QEngine with vLLM, the fastest other engine I found.

Tokens per second, with QEngine’s speedup and 95% interval

QEngine was tuned on these prompts for Qwen3-8B only. On Llama, which used its own sampling settings, it was faster on 54 of 64.

Results with several requests at once are reported for Qwen3.8-27B-FP8 only.

Method

Every engine ran alone on the same GH200 with the same client, prompts and sampling, in the fastest setup I found. The plan was fixed in advance. One later change, faster vLLM setups for many requests, made QEngine’s comparison worse.

QEngine’s code is private, and it has changed since these runs.

Hardware

One NVIDIA GH200 with 96 GB of HBM3, driver 570.148.08, CUDA 12.8. One engine at a time.

Models

Qwen/Qwen3.8-27B-FP8 (Hugging Face revision 017b9c7, 8-bit floating-point weights), Qwen3-8B and Llama-3.1-8B-Instruct.

Prompts

Main: 64 writing prompts of 128–139 tokens from mistral.rs’s v0.9.3 benchmark; QEngine was tuned on them. Code, math and JSON: 24 each, never used for tuning, though earlier builds were measured on them. Long: 16 document-summary prompts at each of about 4,000, 16,000 and 30,000 tokens; QEngine was tuned on them. Many requests at once: 32 prompts from QEngine’s tuning pool.

Output and sampling

512 tokens per request, with the stop signal ignored. Temperature 1, top-k 20, top-p 0.95, min-p 0 and no repetition penalty (these set how randomly each token is picked), one seed for every request, and the model’s chat format with reasoning on. Llama: temperature 0.6, top-p 0.9 and no top-k, in every engine.

Client

vLLM’s benchmark client (vllm bench serve 0.27.1), streaming from each engine’s OpenAI-compatible endpoint.

Other engines

All used speculative decoding. DFlash2 is a published draft model for Qwen3.8-27B-FP8; the number is tokens drafted per step. Multi-token prediction drafts with layers built into the model.

mistral.rs, commit b6ea3f2 (pinned by its v0.9.3 benchmark): DFlash2, 7, the most its draft model allows; shorter lengths were not tested, and lower-precision versions were slower.
vLLM v0.31: DFlash2, 7. With many requests, its fastest setup per load: DFlash2 with 7 at 4 and 8 requests and 3 at 16 and 32, ahead of its multi-token prediction setups.
SGLang v0.5.19.dev99 (commit 1cf2b8c): DFlash2, 7; faster here than its DSpark v2 mode.
TensorRT-LLM v1.3.0rc29: the fastest of five setups fixed in advance (no drafting, multi-token prediction with 2 or 3, DFlash2 with 4 or 7), picked on 16 prompts outside the main set: DFlash2, 7, also used with many requests. This release runs one group of the model’s weights in BF16 rather than FP8.
Qwen3-8B: vLLM v0.31 with EAGLE-3, 5 tokens, the only other engine in the final run; it beat SGLang in setup runs, and mistral.rs has no speculative decoding for this model.
Llama-3.1-8B-Instruct: vLLM v0.31 with DFlash, the fastest of 11 setups fixed in advance across vLLM, SGLang and TensorRT-LLM (one TensorRT-LLM setup), picked on 16 prompts outside the main set.

Runs

Main set and smaller models: a warm-up run, then three measured runs, with each prompt’s median time. Other sets: one run. Many requests at once: one run per load, on one server per engine; every other test had a fresh server.

Speedups and intervals

Each prompt is timed from request to last token. The speedup is the other engine’s total time over all prompts divided by QEngine’s. The 95% interval comes from redrawing the prompts at random 2,000 times (a bootstrap). With many requests at once the measure is total output tokens per second from one run, with no interval.

Where an engine ran faster elsewhere, I compare against that: mistral.rs ran at 98.5% of its published 217 tokens per second here, and SGLang at 99.2% of its earlier run. On October 14 a later search found faster vLLM setups at 4, 16 and 32 requests, which replaced the planned ones and made QEngine’s ratios worse.

Identical output

Code, math, JSON and long sets: compared with the tested build run without speculative decoding. Writing set: compared with a run without it from an earlier build. Qwen3-8B and Llama-3.1-8B-Instruct: 64 of 64.

Setup differences

Context limits differed: the other engines allowed 4,096 tokens per request on the main, code, math and JSON sets, and mistral.rs 4,864 and 17,152 on the ~4,000- and ~16,000-token sets; QEngine’s limit was longer. QEngine and mistral.rs both allowed 32,768 on the ~30,000-token set. The effect was not measured. With many requests, QEngine and vLLM each ran a setup chosen for that test.

Run dates

October 13, 2026: every Qwen3.8-27B-FP8 result at one request, Qwen3-8B, QEngine’s and TensorRT-LLM’s runs with many requests, and vLLM’s at 8. mistral.rs ran back to back with QEngine; the others ran the same day. October 14: vLLM at 4, 16 and 32 requests, compared with QEngine’s October 13 runs, and Llama-3.1-8B-Instruct.