1,844 tokens/s on a 5090. What was actually useful?
I got a Qwen3.8-27B derivative to peak at 1,844 tokens/s on one RTX 5090. Eight streams were generating together, with thinking off. The whole run averaged 1,348 tokens/s and produced 16,669 tokens in 12.4 seconds.
Quite insane to watch. But the result I care about more is this: in a separate test with extra-high reasoning and one active chat, Cinference finished a 64-task coding batch in 3m23s. My Q4 + MTP setup took 11m15s. They passed 60 and 59 tasks respectively.
That makes Cinference worth investigating. It doesn’t establish zero quality loss, and it hasn’t replaced my daily setup. Here’s what I tested and where the comparison stops.
What the video shows
The visible prompt was:
Build an interactive Three.js floating island with an observatory, orbiting planets, airships and waterfalls above the clouds. Make it beautiful, with a day/night toggle.
A custom adapter split the work into 16 scene modules, with up to eight requests active at once. Each request had its own role prompt. A prepared Three.js host assembled the results. OMP displayed the actual output from all active streams as it arrived, with module labels, at normal playback speed.
This matters because “16 modules” doesn’t mean 16 simultaneous generations. It also wasn’t an ordinary coding-agent session independently building an entire application from that one sentence. The decomposition, host and assembly logic were already there.
The model was fafstmobel, a modified Swift/Huihui-derived Qwen3.8 checkpoint, running on Cinference with DFlash2. It is a different target from the original Qwen models in the quality tests below. Those scores don’t transfer to it.
| This take | Result |
|---|---|
| Peak, roughly one second | 1,844 tok/s |
| Whole-run average | 1,348 tok/s |
| Generation time | 12.37s |
| Generated output | 16,669 tokens |
| Active requests | Up to 8 |
| Thinking | Off |
The peak came from the engine’s output-token counter over an approximately one-second logging interval. The average was total output tokens divided by generation wall time. Neither is the speed of one chat.
The revealed scene came from this exact take. It passed browser error and control checks without manual code repairs. The assembler can normalise a returned function declaration, so “no manual repairs” doesn’t mean the responses bypassed all processing. A preceding take reached a higher peak but had a geometry warning; it wasn’t used. This is a selected working demo, not a one-attempt quality benchmark.
The question behind the demo
My normal workload is one or two chats with reasoning_effort: xhigh. I want a finished, useful answer sooner, without quietly giving up the context window or capabilities I already use.
Production was Qwen3.8-27B UD-Q4_K_XL in llama.cpp with MTP, served through llama-swap. It had two chat slots with 204,800 tokens of context each. That allowance includes the prompt, thinking and answer.
I compared four setups on the same RTX 5090:
| Name in the tables | Setup |
|---|---|
| Plain Q4 | UD-Q4_K_XL, llama.cpp, speculation off |
| Q4 + MTP | Same GGUF, llama.cpp, MTP drafting |
| Cinference | NVFP4 target, DFlash2 |
| TensorFold | MLX 4-bit target, DFlash2 |
The pinned builds were llama.cpp b10524-9ee9fc04c, Cinference 383e5db and TensorFold 1.2.7. Plain Q4 means the same quantized model with speculative decoding disabled, not a 16-bit baseline. TensorFold and Cinference use different formats from the GGUF, so this compares complete setups. It cannot isolate the contribution of the engine, quantization or drafter.
The practical idea behind drafting is to propose multiple tokens and verify them with the target model. Whether that helps depends on how much of the proposed output is accepted and what the extra work costs. It doesn’t make a fast result from different weights evidence of identical quality.
Faster tokens can still mean a long wait
The first production latency probes allowed 12,000 output tokens. Both exhausted that allowance on thinking without producing a final answer. The measured decode rate was around 92–93 tokens/s, which was accurate but didn’t describe a useful outcome.
With a 40,000-token allowance, the longer probes finished:
| Setup | Answer 1 | Answer 2 |
|---|---|---|
| Plain Q4 | 505.9s | 494.2s |
| Q4 + MTP | 176.5s | 152.3s |
| Cinference | 83.0s | 62.1s |
These were single-chat, xhigh timing samples with nonempty final answers, not independently graded answers. Output lengths differed substantially: plain Q4 generated about 25,000 tokens per task, while MTP generated roughly 16,200 and 13,800. Faster decoding explains only part of the difference in time to finish.
Cinference decoded at about 265–313 tokens/s on these probes, against roughly 94–95 for MTP and 51–52 for plain Q4. Those rates belong to this workload, not every prompt or context size.
For a quality comparison, I wanted fixed tasks and an actual grader.
Coding quality under the same budget
I used EvalPlus, with its HumanEval+ and MBPP+ tests and official scoring. The full comparison covered 164 HumanEval+ tasks and 378 MBPP+ tasks, one retained sample per task, with xhigh reasoning, temperature zero and seed 271828.
The first comparison retained the historical 2,048-token output allowance, including thinking. That is a tight budget for xhigh. Empty final answers remained failures.
| Setup | Passed /542 | Empty final answers |
|---|---|---|
| Plain Q4 | 396 | 95 |
| Q4 + MTP | 389 | 100 |
| Cinference | 386 | 103 |
| TensorFold | 362 | 150 |
Plain Q4 led on this test, but the difference from Cinference was statistically inconclusive. Cinference’s paired difference was −1.85 percentage points, with a 95% bootstrap interval from −4.61 to +0.92 points. That interval allows both a regression and a small improvement. It is not evidence that the setups are equivalent.
TensorFold’s MLX 4-bit setup had a clearer deficit: −6.27 points versus plain Q4, with an interval from −9.04 to −3.69. This result applies to the tested setup and token budget. It doesn’t identify which component caused the lost passes.
The bootstrap used 20,000 paired draws. These were exploratory comparisons without a multiple-comparison correction or a predeclared equivalence margin.
Giving thinking more room
I also ran a fixed 64-task subset, evenly split between the two suites, with an 8,192-token allowance. It was selected before these results, not assembled from tasks a candidate happened to pass.
The table below uses the later concurrency sweep’s one-chat measurements, which are also the timings used in the thread. Earlier matched runs produced slightly different elapsed times; I haven’t mixed the fastest timing from each campaign.
| Setup | Passed /64 | Batch time |
|---|---|---|
| Plain Q4 | 59 | 16m19s |
| Q4 + MTP | 59 | 11m15s |
| Cinference | 60 | 3m23s |
| TensorFold | 53 | 5m22s |
These are times for the whole batch, including failures, not the latency of one answer. All four used a 10,240-token context cap per request in this sweep. Cinference took about 30% of MTP’s elapsed time, with one more passing answer in this sample.
The larger allowance improved completion, but didn’t remove TensorFold’s lower coding score. It still doesn’t tell us whether quantization, runtime behaviour or token use explains the difference. Nor does a one-task lead on 64 tasks certify Cinference as lossless.
Checks beyond small coding problems
The comparison also included fixed instruction-following, tool-calling and retrieval subsets, using their established scorers.
| Setup | IFEval /200 | BFCL /125 |
|---|---|---|
| Plain Q4 | 178 | 101 |
| Q4 + MTP | 179 | 104 |
| Cinference | 180 | 108 |
| TensorFold | 177 | 103 |
IFEval is strict prompt-level scoring here. BFCL is the selected native tool-calling subset, not the entire benchmark. All four scored 100% on the 120 fixed RULER retrieval prompts at nominal 8k and 32k context. That is useful evidence about those retrieval tasks, not validation of a 204,800-token working conversation.
A fixed 20-task SWE-bench Verified cohort gave a more operational result:
| Setup | Solved /20 | Total agent time |
|---|---|---|
| Plain Q4 | 11 | 157.6 min |
| Q4 + MTP | 15 | 87.0 min |
| Cinference | 15 | 48.9 min |
The accelerated setups solved the same 15 tasks. All four tasks they solved beyond plain Q4 were timeouts for plain Q4. Under a fixed budget, getting through the work faster can improve the success rate without proving better underlying reasoning.
The agent had 80 steps, a nominal 600-second task budget, an 840-second external cap and 8,192 tokens per call. Agent time included environment startup and tool use, but excluded image/model loading and grading. TensorFold wasn’t run on this repository cohort. Twenty tasks are too few to establish general parity.
More chats don’t necessarily finish more work
I separately swept concurrency, keeping the same 64 coding tasks and 8,192-token allowance. For raw throughput, each setting also received three fixed-length trials: 800 output tokens per stream, xhigh, temperature 1 and top_p: 0.95. Every stream had to reach exactly 800 tokens for that trial to qualify.
Those deliberately capped outputs are a speed measurement, not finished answers. The table reports the median of three trials at each setup’s highest measured qualifying setting.
| Setup | Chats | Total tok/s |
|---|---|---|
| Plain Q4, CUDA graphs on | 32 | 499.9 |
| Q4 + MTP | 16 | 388.4 |
| Cinference | 8 | 777.1 |
| TensorFold | 8 | 659.4 |
All totals include thinking. These are separate from the no-thinking video and use different prompts and weights. The 777.1 figure and the video’s 1,844 peak aren’t two estimates of the same thing.
Plain Q4’s best raw setting wasn’t its best coding setting. At 24 chats without CUDA graphs it passed 60/64 in 7m14s. The graph-enabled 32-chat run passed 59/64 in 9m17s. Likewise, MTP’s 16-chat raw maximum didn’t beat its eight-chat coding batch: 60/64 in 4m33s.
Cinference at eight chats completed the coding batch in 1m23s with 59/64 passes. TensorFold at eight took 2m35s with 53/64. Increasing TensorFold’s concurrency from one to eight didn’t change its pass count on this screen.
There were limits and failures too. Plain Q4 couldn’t allocate the tested 48-chat profiles. Its graph-disabled 32-chat quality run failed after request timeouts. MTP failed to allocate at 32, TensorFold refused the 16-chat memory requirement, and this Cinference build has a software limit of eight active requests. These are the best measured settings among those tested, not a proof of the absolute maximum.
Context is the reason I haven’t switched
The throughput sweep capped each request at 10,240 tokens. My daily setup allows 204,800 per chat, twenty times as much. Short prompts don’t exercise that capacity just because a server starts with a large limit.
Memory figures below come from the archived runs, without rerunning inference. “GPU peak” is the highest whole-device sample during work, including about 1 GiB used by other services and driver overhead. It isn’t process-only VRAM or an allocator high-water mark.
| Configuration | Context allocation | GPU GiB |
|---|---|---|
| Plain Q4, graphs, 32 chats | 327,680 total | 29.8 |
| MTP, 16 chats | 163,840 total | 29.8 |
| Cinference, 8 chats | 40,704 shared GPU KV | 27.5 |
| TensorFold, 8 chats | 81,920 total | 28.5 |
Cinference also configured 8 GiB of host KV storage. Its 40,704-token device pool is shared; it doesn’t reserve eight complete 10,240-token contexts on the GPU. TensorFold configured an 8 GiB host prompt cache. Neither figure measures peak total system RAM.
My restored production profile has 409,600 tokens of configured context across two slots. Its recorded idle GPU use was 30.0 GiB, which should not be compared as a workload peak. First-token latency and peak host RAM weren’t measured consistently across the whole campaign, so I haven’t filled those gaps with estimates.
What I’d actually use
I’m keeping Q4_K_XL + MTP as my daily setup for now. Cinference was the most promising tested combination of speed and quality on the original-model workloads, and it’s the next candidate I’d test at my real context sizes.
Before replacing production, I still want evidence for long conversations, two simultaneous large-context chats, the tool and vision paths I use, and stability over sustained use. The 1,844-token/s demo is fun. The 3m23s coding batch is the result that makes that follow-up worth doing.