The September baseline · one model, one GPU, a measured working set.
Short-input generation · Boost
Configured context · 524,288 tokens
RTX 5090 · 32 GB
Historical headline speed: median of three warm runs on September 25, 2026, with short input and 384 output tokens. A 512K allocation does not mean a full 512K prompt runs at this speed.
The answer is a combination: keep the model on the GPU, give memory room to work, verify several proposed tokens at once, and avoid rereading the same conversation.
Explore the measurementsOne workstation.
Three distinct profiles.
LocalAI has grown beyond the original coding setup. The Windows-native 512K llama.cpp profile remains the default. Strata adds a different model for Game Lab and general chat; HyperQwen adds an experimental 64K backend for text coding. These are separate choices, with separate limits.
Native llama.cpp
Qwen3.8-27B Q4_K_M, Q4 KV caches and three-token MTP. The large-context configuration behind the September report is still selected in the source configuration.
Follow the default path ↓Strata
Qwen3.8-Flash-Next IQ3_S with INT8 KV streaming. Game Lab leaves GPU room for previews; general Chat has its own prompt, history and thinking settings.
See the two workflows ↓HyperQwen
Qwen3.8-27B AutoRound W4A16 on vLLM in Docker/WSL2, with DFlash2 speculation. Faster uncached requests in the recorded comparison, with different cache and memory tradeoffs.
Compare the October results ↓Treat the GPU as a shared resource: use one large inference backend at a time, and release it before a competing graphics workload needs the memory. The launcher and Release GPU controls now cover the owned Strata and HyperQwen processes too, with checks for active work. The normal launcher still selects llama.cpp.
New October workflow and backend results sit alongside the original September experiments. The 143.1 tok/s headline above remains a September measurement. The newer client-timed stream rates below use a different measurement method and should not be spliced into a single speedup curve.
The default path: one model, several routes.
The default coding profile in our LocalAI workstation uses Qwen3.8-27B in Q4_K_M format, served by standalone llama.cpp on Windows. A Ryzen 9950X3D and 96 GB of system RAM support an RTX 5090 with 32 GB of GPU memory. The inference runtime is pinned to llama.cpp b11146 with CUDA 13.4.
Goose provides the desktop experience and Pi provides a coding-agent interface. NVIDIA Switchyard sits between clients and inference, choosing request settings and exposing familiar model routes. All five coding routes use the same loaded weights. The router is useful plumbing; it does not make the GPU compute tokens faster.
The coding backend started on project-local Ollama and later moved to standalone llama.cpp. Global Ollama now also serves the separate Decision Lab and synthetic-market research tools described below. It is not a second engine cooperating on the same answer. Running both copies of Qwen at once can consume the memory this configuration needs.
The configured coding routes perform model inference on the workstation, without a cloud-model fallback. Browser research and other network tools still contact external services. A laptop on the LAN can run the client and its tools while the Windows PC performs inference; that does not pool the two machines’ GPU memory.
First, make the working set fit.
Fast generation is difficult when a model competes with a game, another model, or its own oversized working buffers. Memory planning came before chasing another token per second.
Compress what the model knows.
Q4_K_M stores weights at mixed, reduced precision. The local GGUF file is 16.81 GB; the recorded CUDA model buffer is 15,339 MiB. The runtime reports 66/66 offloadable layers on the GPU.
The bit diagram is conceptual. Q4_K_M is not uniformly four bits per parameter; file size also includes metadata and other tensors.
Compress what it has just read.
The KV cache holds attention state for processed tokens. Its precision is independent of weight precision. For 512K, we selected Q4 K and V caches: 9,216 MiB for the main cache, with a 576 MiB MTP draft cache in the recorded runtime.
Qwen uses a hybrid architecture. A textbook full-attention memory formula applied to every layer would misrepresent this model.
Flash Attention uses a GPU-friendly attention implementation that reduces memory traffic. Smaller physical batches reduce scratch-buffer pressure. We retained a logical batch of 2,048 but lowered its physical microbatch to 128, used Q4 draft caches, and moved the vision encoder to the CPU. The image encoder tradeoff is slower possible image preprocessing in exchange for GPU headroom.
The actual GPU budget is tight
PEAK TOTAL DEVICE MEMORY · MiBThe first 512K attempt generated only 34.4 tok/s on short input. The selected memory configuration reached 143.1 tok/s. Windows shared-GPU-memory usage also fell, from roughly 3.3 GB to 0.4 GB. That is consistent with lower memory pressure, but shared usage alone does not prove paging.
The practical rule is simple: give this large-context profile the GPU. The launcher avoids loading a duplicate idle Qwen copy and retains a healthy, matching server. Release GPU mode unloads the owned model after work. No overclock, BIOS change, voltage adjustment, or power-limit modification was part of the solution.
Propose ahead.
Let the main model check.
Normally, a language model produces one token after another. Multi-Token Prediction, or MTP, lets prediction heads associated with this model propose several upcoming tokens. The main model verifies the proposals. Accepted tokens advance the answer; rejected proposals are discarded or corrected.
Our Boost profile allows three draft tokens. When enough proposals are accepted, verification can do more useful work per step. This is different from routing easy prompts to a separate small model. MTP still needs memory and computation, and its advantage depends on the workload and acceptance rate.
A speculative step, illustrated
The draft heads propose three continuations after “return”. These are candidates, not final output.
More output tokens per second with MTP in our two matched 512K speed workloads. First-content latency was slightly higher because MTP added prompt-processing work.
Original → optimized.
Here are the numbers.
Our first baseline vs. the selected September setup
SHORT-INPUT WARM MEDIANSOllama defaults · 128K allocated context
Temperature 0.2 · 3 warm samples
llama.cpp Boost · 512K allocated context
Greedy sampling · 3 warm samples
| Metric | Original | Optimized | Change |
|---|---|---|---|
| Generation speed | 77.6 tok/s | 143.1 tok/s | +84.3% |
| Whole request | 5.58 s | 3.54 s | 36.7% shorter |
| First content | 0.636 s | 0.858 s | 34.9% longer wait |
| Allocated context | 131,072 tokens | 524,288 tokens | 4× capacity |
Same Qwen3.8-27B Q4 model and workstation; approximately 1.6K input tokens and 384 generated tokens. These are historical endpoints, not a controlled test of one change: engine build, sampling, cache settings, context size and GPU conditions differ. The baseline's actual request records show 128K allocation; its pre-run inventory showed 32K. The fourfold context increase is capacity, not a fourfold increase in prompt length during this speed test.
Which part of the optimization produced which gain?
| Experiment | Before | After | Interpretation |
|---|---|---|---|
| MTP at 64K · September 13 | 79.5 tok/s | 149.1 tok/s | +87.5%; matched Standard vs. Boost after Unreal was closed |
| MTP at 512K · September 25 | 80.5 tok/s | 143.1 tok/s | +77.7%; same selected memory settings, MTP off vs. on |
| 512K memory tuning | 34.4 tok/s | 143.1 tok/s | 4.16× throughput; MTP on in both, several memory settings changed |
The 34.4 tok/s figure is the first attempt at the much larger 512K configuration. It is not the original workstation baseline. The 149.1 tok/s result belongs to the earlier 64K setup; it is not the September 25 result at 512K.
Read the comparisons using three different clocks.
First content
The wait until visible answer text begins. New input has to be read first.
Decode speed
How quickly output tokens are generated after prompt processing.
Whole request
The wall time you experience, including input processing and generation.
Standard vs. Boost
0.80 s0.86 s
5.55 s3.54 s
September 25: same selected 512K settings, three warm samples per input size and profile, greedy sampling, 384 output tokens. Cached input excluded. Blue = Standard; green = Boost. Each displayed value is its own median, so components need not sum exactly.
For 13.2K input, Boost cut median whole-request time from 10.91 to 9.01 seconds, even though generation became about 81% faster. Why the smaller end-to-end gain? The request still spends around six seconds reaching its first content. Faster output cannot erase the cost of reading input.
The 512K tuning path
SHORT-INPUT DECODE · TOK/SGPU contention could erase the benefit.
With changing Unreal Engine and desktop activity, the old 64K Ollama Boost run measured 64.6 tok/s, compared with 75.7 for Standard. After Unreal was closed, Boost measured 149.1 and Standard 79.5 tok/s. The shared-GPU runs had changing conditions, so their ratios do not isolate MTP. They do explain why we treat GPU ownership as part of the system.
The biggest shortcut is
not reading it twice.
The model card specifies 262,144 native context tokens. We configured 524,288 using YaRN with factor 2 and the original 262,144-token context. YaRN changes positional scaling; it does not create free memory or guarantee reasoning quality. The Q4 caches and memory adjustments made the larger allocation practical on this GPU.
Context includes the prompt, conversation, tool definitions, reasoning, and answer. A “512K” setting is not 512K of documents plus unlimited output. We disabled context shifting so oversized requests fail instead of silently dropping history, and the coding client reserves headroom for compaction and output.
Did it actually read beyond 262K?
497,969 INPUT TOKENSSame long conversation, different wait
About 10.2 minutes to process a new 497,969-token input.
The cached request contains 498,042 input tokens, of which 498,017 were reused (99.995%). Blocks are illustrative, not a token-level trace.
The repeat began answering in 0.803 seconds because almost all its input had already been processed. This is prompt-prefix reuse, not a universal 760× faster model. A subsequent 384-token code speed probe at about 498K history decoded at 39.3 tok/s—much slower than the short-input headline, and not graded for code correctness.
The engine keeps up to 16,384 MiB of prompt cache in system RAM and uses eight recurrent checkpoints with 65,536-token spacing. These support reuse when conversations change. They are separate from GPU KV allocation, are not model training, and are not a durable memory of every past chat. Keep the server loaded and the prefix stable; retrieve relevant file ranges instead of filling the window just because it exists.
Removing the wrapper
was a small part of the story.
We tested Ollama against direct requests to the very same persistent llama.cpp child process. Same weights. Same GPU placement. Same 128K allocation and Q8 cache.
+2.3% median generation speed
+0.4% median generation speed
Five warm samples per path and prompt size, 384 output tokens each, with path order alternated. Retained samples had zero cached input. The sample ranges overlap; only 3 of 10 paired outputs were byte-identical, and MTP acceptance varied with the text. Small differences cannot be cleanly assigned to HTTP overhead.
The migration gave us direct control over context extension, independent cache precision, batching, and speculative settings. It did not introduce Flash Attention or MTP for the first time. The older 128K setup was faster on these small prompts than the selected 512K setup. We chose more context and control, not a demonstrated across-the-board speed increase.
Faster requests.
A different set of tradeoffs.
HyperQwen gave us a reason to test another backend. We compared the existing 512K llama.cpp profile, a 64K llama.cpp control, and the experimental 64K HyperQwen configuration on the same RTX 5090.
The control matters: reducing llama.cpp’s allocation from 512K to 64K already improved these request times. Only context and the service port changed; the control retained the original YaRN setting. A client history limit alone would not have made that change.
HyperQwen changes more than the engine. This pinned experiment uses vLLM 0.31.0, AutoRound W4A16 weights, BF16 KV state, FP16 recurrent state and seven-token DFlash2 speculation. Native llama.cpp uses GGUF Q4_K_M, Q4 caches and three-token MTP. This is a comparison of complete configurations, not an engine-only comparison.
Uncached request timing
384 output tokens · temperature 0 · thinking off · no cached input
1.6K input · whole-request time
Seconds · shorter is faster · scale 0–5 s
| Configuration | First content | Whole request | Approx. stream rate |
|---|---|---|---|
| llama.cpp 512K | 1.716 s | 4.704 s | 128.7 tok/s |
| llama.cpp 64K control | 0.843 s | 3.045 s | 174.3 tok/s |
| HyperQwen 64K | 0.417 s | 1.546 s | 340.3 tok/s |
13K input · whole-request time
Seconds · shorter is faster · scale 0–18 s
| Configuration | First content | Whole request | Approx. stream rate |
|---|---|---|---|
| llama.cpp 512K | 12.586 s | 16.137 s | 108.3 tok/s |
| llama.cpp 64K control | 5.941 s | 8.571 s | 146.0 tok/s |
| HyperQwen 64K | 3.209 s | 4.584 s | 279.3 tok/s |
47K input · whole-request time
Seconds · shorter is faster · scale 0–48 s
| Configuration | First content | Whole request | Approx. stream rate |
|---|---|---|---|
| llama.cpp 512K | 41.326 s | 44.493 s | 113.2 tok/s |
| llama.cpp 64K control | 23.264 s | 26.312 s | 121.9 tok/s |
| HyperQwen 64K | 13.160 s | 14.906 s | 218.3 tok/s |
Three retained samples per size and configuration; input token counts match across configurations for each repeat. Initial model loading/compilation and two warmup requests are excluded. HyperQwen’s initial startup took about five minutes. Each input size has its own chart scale.
On these uncached workloads, HyperQwen reduced whole-request time by roughly 3.0–3.5× relative to the 512K profile, and 1.8–2.0× relative to the 64K control. That is useful evidence for this workload, not a prediction for every coding session. The stream rate uses client arrival timestamps; speculative chunks make it approximate. It is not the native decode metric used in the September charts.
The follow-up tells a different story.
A separate cached-prefix probe favored llama.cpp. All three answered correctly, but HyperQwen reused fewer tokens. Faster cold input did not guarantee the shortest wait on a continuing conversation.
| Configuration | Cold input | Cached follow-up | Reused / input tokens |
|---|---|---|---|
| llama.cpp 512K | 10.105 s | 0.176 s | 13,247 / 13,265 |
| llama.cpp 64K control | 6.107 s | 0.129 s | 13,247 / 13,265 |
| HyperQwen 64K | 3.298 s | 0.224 s | 12,544 / 13,265 |
Speed was only one check.
| Configuration | Coding tasks | Assertions | Peak GPU memory |
|---|---|---|---|
| llama.cpp 512K | 10/12 | 62/64 | 31,877 MiB |
| llama.cpp 64K control | 10/12 | 62/64 | 19,447 MiB |
| HyperQwen 64K | 11/12 | 63/64 | 24,978 MiB |
HyperQwen fixed the transaction case that both llama.cpp profiles missed. All three still failed a Rust decimal error-priority case. One additional pass in twelve previously used fixtures is not a broad quality ranking. Every profile passed the API tool-call round trip; separate Pi JavaScript and Rust repair smoke checks passed on both backends.
The test was text-only and bounded to 64K HyperQwen. It did not test vision, 512K use, concurrency scaling or long unattended reliability. HyperQwen’s Goose routes passed, but its desktop GUI was not exercised. The 512K default configuration was restored after the experiment and remains selected today.
The request audit matched 23 non-model payload pairs and validated the uncached, fixed-output speed samples. The 64K control passed the same matching and contention checks. An earlier 512K attempt was interrupted when another client loaded Ollama; that entire attempt is excluded. Reported GPU peaks include desktop allocations.
Fast text is only the engine.
The surrounding tools make the model useful. They also determine how much irrelevant input, repeated work, and waiting each task creates.
Choose a route
coding uses direct answers and greedy sampling. coding-think enables medium reasoning. coding-long retains a compatible thinking alias; coding-auto experimentally switches reasoning modes.
Find the right context
Project Knowledge uses CPU-based SQLite FTS5 keyword search across registered code and documents, then bounded source reads with references. It loads no embedding model onto the GPU. Explicitly saved project notes survive new chats.
Act and verify
Goose and Pi expose file, editing, shell, and test tools. The real loop is read → edit → execute → inspect. Windows desktop tools prefer accessibility information where possible, avoiding a screenshot on every step.
Research beside the task
The browser extension adds search, source-linked reading, and an adjacent browser. Its agent cursor is independent of the Windows pointer. Web tools use the network; local inference does not make the whole workflow offline.
Build actual applications
Electrobun Apps supplies native web-based application windows with WebGL2 and WebGPU. Rust App Lab supports Cargo builds, tests, checkpoints, and release packages. These are tool capabilities, not gains in raw inference speed.
Keep sessions warm
The launcher reuses a healthy server with matching configuration. Switching Standard/Boost restarts it and clears prompt reuse; selecting the existing profile retains it. Release the GPU when another graphics workload needs it.
coding-compact gives the client a 65,536-token history budget and earlier compaction. It still uses the same physical 512K engine. The route does not shrink VRAM usage or load a smaller model.
Strata: a game workspace and a separate chat.
Leave room for the preview.
Strata serves Qwen3.8-Flash-Next IQ3_S, a different model from the native 27B default. Its 128K profile uses INT8 KV streaming and reserves 4,096 MiB of VRAM for preview workloads. About 3.6 GiB remained free after loading in the recorded check. This reserve differs from the benchmark profile and can cost inference speed; large Unreal scenes can still need the model released.
Game Lab starts with reasoning off and exposes native web-game, Godot and Unreal tools. A recorded Goose session repaired two game-logic files, passed seven Node checks and a Godot headless check, and exercised all five MCP servers. UnrealBuildTool queried targets; no full Unreal game build or rendered playtest was performed.
Give thinking a budget.
The separate Strata Chat profile uses the same model with a general-purpose prompt and its own chats. It has no Game Lab instructions or development tools. High thinking is capped at 8,192 tokens within a 16,384-token total response budget, leaving room for the answer; temperature is 0.7.
Both profiles have a 131,072-token context budget and compaction at 75%. Saved briefs, progress notes and handoffs carry work between sessions. Compaction alone does not establish reliable overnight autonomy. The accepted Game Lab session reached about 43K input; maximum context and repeated compaction have not been stress-tested.
Local models now feed small research tools, too.
Ask for a bounded decision.
A local Electrobun workbench tests log triage and deterministic NPC actions. Clef returns choice distributions, yes/no probabilities and ordinal scores. Qwen returns schema-constrained labels, booleans and integers without invented probabilities. Model probabilities are estimates, not calibrated correctness.
The recorded 12-case smoke set gave Clef 27B 11/12 and Qwen 9/12. Clef Flash 9B failed to run in that environment, so it has no valid timing or quality result. This small authored set is not a general model ranking. Suggested log commands are never executed.
Generate once. Execute deterministically.
Qwen proposes bounded numerical strategies; the host validates their syntax tree and translates them into a restricted Rust virtual machine. The simulation then runs each tick without model calls. It uses synthetic markets and paper accounts, with separate development, selection and final-test markets.
The recorded search promoted no strategy over the cash baseline. That negative result belongs in the report. This is a software and research experiment, with no broker integration or claim of real-market profitability. These labs use separate Ollama profiles, not the 512K coding server.
Responsiveness also comes from doing less unnecessary work: use direct mode for routine edits, reasoning mode when it earns its cost, small tool results, and relevant source ranges. None of those choices guarantees correctness. Automatic chat naming remains enabled; convenience features were not silently sacrificed to make the numbers look better.
Measure speed.
Keep the failures.
The September results below and the October comparison above come from saved local experiments. Configuration and source records were reviewed on October 28; the inference benchmarks were not rerun for this update. A configured feature, a recorded integration pass, and a tested everyday workflow are different kinds of evidence.
| Check | Recorded result | Scope |
|---|---|---|
| September native routing | 5/5 routes | Mock translation and live reasoning smoke checks |
| Native tools + image input | Passed | Function-call round trip and a simple two-color image |
| Pi JavaScript / Rust repairs | 7.35 s / 7.61 s | Two small fixtures; tests passed and remained unchanged |
| Near-500K retrieval | 3/3 values | One seeded case and its cached repeat; not arbitrary long-context reasoning |
| Earlier core coding suite | 9/9 tasks · 47/47 groups | Used during tuning; a regression set, not a blind score |
| Earlier fresh holdout | 2/3 tasks · 16/17 groups | Rust error-priority failure; later repair does not change this score |
| Earlier strict tool fixture | Failed | Repeated reads or prose outside the required JSON contract |
We selected Qwen3.8 Q4 for a balance of speed and observed task success. In the initial common-sampler comparison it passed 17/18 attempts; faster sparse-model candidates passed fewer. This was not an exhaustive per-model optimization study. A fast wrong answer is not a completed coding task.
Q4 weights, Q4 KV state, and extended positional scaling each carry quality tradeoffs. Small synthetic tests and two successful repairs do not establish reliable autonomy on large repositories. Full-context generation is slower, cold inputs are expensive, and 512K Boost leaves little GPU headroom.
Selected configuration, without machine-specific paths
Model Qwen3.8-27B Q4_K_M
Runtime llama.cpp b11146 · CUDA 13.4
Context 524288 · YaRN factor 2 · original 262144
GPU layers all · runtime reported 66/66 offloaded
Attention Flash Attention on
KV cache K=q4_0 · V=q4_0
MTP draft-mtp · up to 3 draft tokens
Draft cache K=q4_0 · V=q4_0
Batch logical 2048 · physical 128
Parallel 1 active request
Image encoder CPU
RAM cache 16384 MiB
Checkpoints 8 · minimum spacing 65536 tokens
Context shift disabledThese are this workstation’s selected settings, not a portable recommendation for every GPU. Standard preserves context/cache settings while disabling MTP.
How to make a comparison trustworthy
- Record the exact setup. Model digest, engine version, context allocation, sampling, batch and cache settings, GPU occupancy, and runtime arguments.
- Control the workload. Use the same input class and output cap. Separate a warm model from a reused prompt: they are different kinds of “warm”.
- Check all three clocks. Report first content, decode tokens/sec, and complete wall time. Do not turn an output-rate gain into an end-to-end claim.
- Audit reuse and output. Count cached and new tokens; retain raw responses, stop reasons, GPU samples, and incomplete runs.
- Test the actual task. Execute independent tests after code edits. Keep holdout and tool-contract failures alongside successful fixtures.
Measurement provenance
The two accompanying data files contain selected numeric summaries, source filenames and SHA256 fingerprints. They exclude raw conversations, credentials, usernames and network addresses. The September file is unchanged; the October file records the new comparison and configuration review. The underlying private logs are not published here.
September measurement data · JSONOctober update data · JSONLocal experiment records
- September 13: initial tuning, shared-GPU runs and coding fixtures.
- September 25: controlled wrapper comparison, 512K memory tuning, MTP comparison, near-500K retrieval and native integration checks.
- September 29: original article review of configuration, launchers and tool documentation.
- October 3–7: Decision Lab, synthetic-market research and Strata workflow integration records.
- October 11: audited three-configuration HyperQwen comparison, cache and coding checks.
- October 28: source/configuration review for this update; no new inference benchmark.
Primary technical references
- HyperQwen upstream project · local experiment pinned to revision
3512f0e; upstream defaults may differ. - Qwen3.8-27B model card and context guidance
- llama.cpp b11146 release
- Pinned server options and prompt caching
- MTP speculative-decoding implementation
- FlashAttention: the original paper
- Ollama v0.34.4 llama-server wrapper
- Weight quantization · KV caching
Optimize the work
around the model, too.
A well-fitted GPU working set makes generation viable. MTP accelerates the output phase. A stable prompt cache makes follow-up turns responsive. Focused tools reduce how much the model must read. Tests tell us whether any of that speed produced a useful result.
The October experiments add another lesson: choose a configuration for the work in front of it. More context, faster uncached output, room for game previews and reliable tool use are separate requirements. Keep the measurements attached to the configuration that produced them.
Revisit the system map