# Qwen3.8-27B VRAM Guide: RTX 5090, Dual GPUs, and 262K Context

> A 32GB RTX 5090 can run quantized Qwen3.8-27B, but fitting 262,144 tokens depends on the weights, KV cache, runtime, and workload. Use these memory budgets and configuration checks to choose a practical single- or dual-GPU setup.

- URL: https://blog.laozhang.ai/en/posts/qwen3-8-27b-vram-guide
- Published: 2026-09-06
- Updated: 2026-09-06
- Author: LaoZhang AI Team (https://blog.laozhang.ai/en/about)
- Topic: Developer Tools & Agents
- Tags: Qwen3.8, Local LLM, RTX 5090, VRAM, Quantization

---
**For a single RTX 5090, start with quantized weights, one request at a time, and a 32,768-token context. Treat 262,144 tokens as a separate configuration to validate.** Four-bit weights make the model much smaller, but they do not automatically make its attention cache four-bit. That distinction can change a full-context memory estimate by more than 10 GiB.

Qwen3.8-27B has a native maximum context of **262,144 tokens**, shared by the input and generated output. It is a dense vision-language model with a hybrid attention architecture, so both conventional “parameters times two” estimates and generic Transformer KV-cache calculators miss important details. The model's [official configuration](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json) supplies the dimensions needed for a more useful estimate.

The practical choice is between a straightforward GGUF deployment with explicit cache settings, a checkpoint-specific NVFP4 serving setup, and two GPUs with a runtime that distributes the model. Below, file sizes and calculations are separated from configurations reported by runtime maintainers and independent users. None of those reports establishes performance on every machine with the same GPU name.

## Start with the weights you will actually load

The desktop [RTX 5090 has 32GB of GDDR7 and no NVLink](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/). Check the capacity and available memory reported by your driver before making a budget; a GPU running a display may already have allocations.

```bash
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free --format=csv
```

Keep units consistent. Decimal **GB** means 1,000,000,000 bytes; **GiB** means 1,073,741,824 bytes. Convert the driver's reported units before subtracting the figures below. The marketed “32GB” designation should not replace the actual memory reading.

| Weight format or checkpoint | Downloaded weight size, GB | Size, GiB | What it means for planning |
| --- | ---: | ---: | --- |
| Official BF16 | 55.563 | 51.75 | Cannot reside entirely on one 32GB card |
| Official FP8 | 30.867 | 28.747 | Leaves a narrow single-card budget before cache and runtime allocations |
| Unsloth UD-IQ4_XS GGUF | 14.253 | 13.274 | Smaller weight allocation to investigate when memory is tight |
| Unsloth UD-Q4_K_M GGUF | 16.464 | 15.334 | A useful starting point for a single-card budget |
| Unsloth UD-Q5_K_M GGUF | 19.772 | 18.414 | More weight memory to balance against context |
| Unsloth UD-Q6_K GGUF | 21.984 | 20.474 | Less room for long-context cache on one card |
| Unsloth Q8_0 GGUF | 29.047 | 27.052 | Long context quickly makes a second GPU or offload relevant |

The BF16 figure is tensor storage from the [official weight index](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/model.safetensors.index.json), excluding small file headers. FP8 is the sum of the [official safetensors files](https://huggingface.co/Qwen/Qwen3.8-27B-FP8/tree/main). GGUF figures refer to [Unsloth revision `4ca720788d1e01f1bff70c033e0d0028fd02e502`](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/4ca720788d1e01f1bff70c033e0d0028fd02e502), checked September 6, 2026. These are storage measurements, not measured resident VRAM.

Two consequences matter when choosing a download. First, official FP8 is not exactly half the BF16 size: scales and components retained at other precisions affect the result. Second, a label such as “Q4” does not identify one universal checkpoint. Publisher, quantization scheme, mixed precision choices, and included components matter. Do not substitute an unrelated four-bit file into a memory or quality claim about `UD-Q4_K_M`.

GGUF and NVFP4 also require compatible loading paths. A vLLM NVFP4 recipe is not a set of switches that can be applied to an arbitrary GGUF file. Choose the runtime and supported checkpoint together, then adjust context and cache precision within that combination.

Choose weight precision with a small set of your real tasks. If a four-bit checkpoint gives acceptable answers, spend the remaining memory on the context and concurrency you need. If it fails important tasks, compare a higher-precision checkpoint while keeping the cache settings and inputs fixed. File size alone cannot rank answer quality, and changing both weight and cache precision at once makes it harder to identify the cause of a regression.

## Why the KV cache changes the answer

Qwen3.8-27B has 64 layers, but only **16 use full attention**; the other 48 use linear attention. Each full-attention layer has four KV heads with a head dimension of 256. Applying the ordinary growing KV-cache formula to all 64 layers would overestimate this part of memory fourfold. The linear-attention layers still need recurrent state; they do not cost zero memory. These dimensions come from the [model configuration](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json).

For one sequence, the full-attention KV payload at F16 or BF16 is:

```text
2 × 16 × 4 × 256 × 2 = 65,536 bytes per stored token
│   │    │    │    └─ bytes per value
│   │    │    └────── head dimension
│   │    └─────────── KV heads
│   └──────────────── full-attention layers
└──────────────────── K and V
```

That is **64 KiB per token**, or 16 GiB at the native maximum. This arithmetic covers the full-attention payload only, assuming both K and V use the stated precision.

| Stored tokens in one sequence | F16/BF16 KV | Ideal FP8 KV | llama.cpp q8_0 KV | llama.cpp q4_0 KV |
| ---: | ---: | ---: | ---: | ---: |
| 32,768 | 2 GiB | 1 GiB | 1.0625 GiB | 0.5625 GiB |
| 65,536 | 4 GiB | 2 GiB | 2.125 GiB | 1.125 GiB |
| 131,072 | 8 GiB | 4 GiB | 4.25 GiB | 2.25 GiB |
| 262,144 | 16 GiB | 8 GiB | 8.5 GiB | 4.5 GiB |

FP8 and `q8_0` are not interchangeable labels. The [llama.cpp block definitions](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-common.h) allocate 34 bytes for 32 values in `q8_0` and 18 bytes for 32 values in `q4_0`. Their scales explain why the payload is slightly larger than ideal eight-bit or four-bit storage. Allocator overhead and runtime-specific padding come afterward.

This gives a revealing single-card example. With Unsloth `UD-Q4_K_M` and 262,144 stored tokens:

```text
F16 KV:   15.334 + 16.0 = 31.334 GiB
q8_0 KV:  15.334 +  8.5 = 23.834 GiB
q4_0 KV:  15.334 +  4.5 = 19.834 GiB
```

The F16 case leaves too little room for ordinary runtime needs on a 32GB RTX 5090. The quantized-cache cases have more plausible headroom, but neither is a fit guarantee: the first term is still a weight-file estimate, and the sums omit recurrent state, activations, workspaces, CUDA graphs, and optional features. Quantized KV also needs task-quality testing; a successful allocation does not establish that cache compression preserves the answers you need.

![Calculated weight and full-attention KV-cache subtotals for UD-Q4_K_M at 262,144 tokens with F16, q8_0, and q4_0 cache.](https://blog.laozhang.ai/posts/en/qwen3-8-27b-vram-guide/img/qwen3-8-27b-kv-cache-memory-budget.webp)

### Budget for the peak, including optional features

A more complete planning equation is:

```text
Peak VRAM = resident weights
          + full-attention KV cache
          + linear-attention recurrent state
          + activations, workspaces, graphs, and allocator overhead
          + optional vision and speculative-decoding allocations
```

The [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B) reports approximately 153.9 MB for one GDN state slot in FP32 or 78.4 MB in BF16. The runtime's reserved slot count and speculative intermediates determine the total. A fixed-size state per slot is helpful for long-context scaling, but it remains an allocation you must account for.

For GGUF vision inference, the matching `mmproj-F16.gguf` in the [same Unsloth snapshot](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/4ca720788d1e01f1bff70c033e0d0028fd02e502) adds 927,607,488 bytes, about 0.864 GiB, of projector weights. Image-processing buffers add further memory. A text-only fit therefore does not establish a multimodal fit.

Concurrency changes the budget again. Two unrelated sequences with 131,072 cached tokens each require the same full-attention payload as one sequence with 262,144 tokens, before their additional per-sequence costs. Prefix sharing may help when a runtime actually reuses shared tokens, but it should not be assumed for unrelated requests. Multi-token prediction (MTP), used for speculative decoding, adds its own allocations and compatibility requirements; enable it after the basic workload is stable.

## A practical first configuration for one RTX 5090

For a text-only GGUF deployment, begin with `UD-Q4_K_M`, a 32,768-token context, and one active sequence. At this length the calculated weight-file size plus `q8_0` full-attention cache is about **16.397 GiB**, leaving substantially more room to discover actual runtime overhead than a full-context first launch.

The following starting configuration adapts flags from the [llama-server documentation](https://github.com/ggml-org/llama.cpp/tree/master/tools/server). It requires validation on your hardware and installed build. Replace the model path with your downloaded file and check your server's help for model support and flag availability.

```bash
llama-server --version
llama-server --help

llama-server \
  --model /path/to/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0
```

The high GPU-layer request is intended to offload all available model layers; confirm what actually loaded from the startup log. Keep the context at 32,768 until a representative request completes. Then increase it to 65,536, 131,072, and finally 262,144, repeating the long-input checks below. Changing one variable at a time makes a failure easier to attribute to cache capacity, prefill memory, or a particular quantization setting.

If you prefer vLLM, its [Qwen3.8-27B recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-27B) includes a **single-RTX-5090 Inferact NVFP4** example with tensor parallelism set to one, `--max-model-len 32768`, FP8 KV, `--enforce-eager`, and `--reasoning-parser qwen3`. The maintainers report a graph-capture out-of-memory failure without eager execution in that tested setup. Use the checkpoint and version stated in the recipe; eager execution is a documented response to that failure, not a universal requirement for every NVFP4 deployment.

### What the single-card 262K report actually establishes

The [MiaAI-Lab single-5090 configuration](https://github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090) reports a 262,144-token setting using RadixArk NVFP4, `vllm==0.27.1` plus a backport of change `#40914`, `turboquant_4bit_nc` cache, a 5.5 GiB KV pool, one sequence, a 512-token batch limit, and MTP3.

That configuration is worth investigating if you can maintain the specified software combination. It is not evidence that a stock vLLM install, another four-bit checkpoint, or the 32K baseline above will reach the same capacity unchanged. The repository also describes garbled output with the stock path and crashes involving MTP with concurrency. Check those constraints before adopting it as a service configuration.

A reported maximum context and a reported decode rate must also describe the same workload before they can support a full-context performance claim. A speed figure from a shorter request cannot tell you how long a 262K prefill will take or what decode performance follows it.

## When a second GPU is the better tradeoff

Two GPUs become useful when you want less aggressive weight quantization, higher cache precision, more simultaneous requests, or enough margin to avoid a highly specialized single-card setup. They do not create one automatically pooled memory space, and they do not imply twice the speed.

At full context, `Q8_0` weights plus F16 full-attention KV total approximately **43.052 GiB** before other allocations. That exceeds one 32GB card but is a plausible aggregate starting budget for two such cards. Original BF16 weights plus the same cache total approximately **67.75 GiB**, already exceeding two nominal 32GB cards before overhead. A second card therefore does not remove the need to choose precision deliberately.

Two 24GB cards may appear adequate for the 43.052 GiB subtotal, but aggregate arithmetic cannot establish that each GPU's assigned layers, cache, buffers, and free memory fit. An uneven split or a larger allocation on the main GPU can still fail.

### Choose how the runtime distributes work

For GGUF, llama.cpp documents `--split-mode layer` as splitting layers and KV across GPUs. You can extend the single-card command with:

```bash
--split-mode layer --tensor-split 1,1
```

The proportions request an equal distribution; they do not certify equal peak allocation. Check both GPUs while processing an actual prompt. For cards with different free memory, adjust the proportions to leave room for each device's other allocations. The documented `row` mode handles KV and intermediate results differently, with these on the main GPU, so do not reuse a layer-split budget without checking the [runtime's split-mode behavior](https://github.com/ggml-org/llama.cpp/tree/master/tools/server).

For vLLM, the upstream recipe uses **tensor parallelism of two on two RTX 5090s**, distributing computation within the model. Its tested build is `0.26.1rc1.dev608+g99a10304d`; the broad minimum-version heading is not the same as that exact tested environment. The [reported startup results](https://recipes.vllm.ai/Qwen/Qwen3.8-27B) are:

| Checkpoint in the TP2 recipe | Reported weights per GPU | Reported KV-pool capacity |
| --- | ---: | ---: |
| Official FP8 | 14.28 GiB | 377,456 tokens |
| Inferact NVFP4 | 12.02 GiB | 445,875 tokens |
| Unsloth NVFP4 | 10.64 GiB | 920,517 tokens |

These token counts describe the startup cache pool, which can serve token storage across requests. They do not raise the model's native per-request context limit, and they are not proof of a successful prefill at every length represented by the pool. The weights-per-GPU measurements also belong to this runtime and checkpoint combination; they should not be substituted for GGUF residency estimates.

Because the RTX 5090 has no NVLink, inspect your PCIe arrangement before choosing tensor parallelism for performance. Use `nvidia-smi topo -m` to see the reported topology, then benchmark your actual workload. Capacity alone cannot predict the communication cost or whether layer splitting and tensor parallelism will behave similarly on your motherboard.

CPU offload is another capacity option when the available GPU memory is insufficient. It changes where memory must be available and may move work or transfers off the GPU; it does not make a partly offloaded result comparable to a fully resident GPU benchmark. Record offload alongside the checkpoint and context when comparing configurations.

## Verify that the requested context is usable

“262K” is a shorthand for the model's **262,144-token** limit; “256K” in binary counting can refer to that same number. The number you set at launch is only the beginning of validation. Distinguish these four observations:

1. **Configured:** the server accepts the requested maximum context setting.
2. **Allocated:** its startup log reports sufficient cache capacity for the intended requests.
3. **Processed:** a tokenized long input completes prefill and generation without truncation or an out-of-memory error.
4. **Useful:** the output correctly uses relevant information throughout that input.

![Four stages for validating long-context inference, with output-token reserve and checks across the beginning, middle, and end of the input.](https://blog.laozhang.ai/posts/en/qwen3-8-27b-vram-guide/img/qwen3-8-27b-long-context-validation.webp)

Reserve output space inside the limit. For example, a total input of 258,048 tokens leaves 4,096 tokens under 262,144. That input count must already include system instructions, chat-template tokens, tool definitions, and any other material passed to the model. The file's word count or character count is not a tokenizer measurement.

Create a test document with distinct, verifiable facts near its beginning, middle, and end. Ask questions that require those facts, plus at least one answer that depends on combining distant sections. Record the actual token count, whether truncation is enabled, time to first token, generation rate, and peak memory on every GPU. Run this with the output allowance and concurrency you intend to use, rather than reducing them only to pass the test.

Compare the resulting answers against a shorter context or a higher-precision configuration using the same facts. This helps distinguish “the server stayed alive” from “the compressed cache still supports the task.” Add images and MTP separately if you need them, and repeat the representative workload after each addition.

The distinction matters even when following official runtime documentation. The [SGLang consumer configurations](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B) were validated with an 8,192-token input, a 1,024-token output, and concurrency one. Those conditions are useful reproducibility information; they are not a 262K workload test.

## Diagnose failures by when they occur

| Failure point | What to inspect | Useful next step |
| --- | --- | --- |
| Loading weights | Exact checkpoint, resident weight allocation, free VRAM | Choose smaller compatible weights, change the device split, or consider offload |
| Cache allocation at startup | Context setting, cache dtype, reserved sequence capacity | Reduce context or concurrency; verify that the intended cache dtype was accepted |
| CUDA graph capture | Runtime version and graph allocations | Check the matching recipe; try eager execution where the runtime supports it |
| Long-input prefill | Batch size, temporary buffers, vision input, peak memory | Reduce the prefill batch setting using runtime-supported options and retest the same input |
| Second simultaneous request | Active token total, state slots, MTP interaction | Reproduce with one request, then budget and test concurrency explicitly |
| Garbled or unreliable answers | Checkpoint/runtime compatibility and weight/KV precision | Return to a documented compatible setup and compare a known-answer task |

For an existing single RTX 5090, the most informative next step is a conservative quantized launch followed by measured context increases. For a hardware purchase, first decide whether your requirement is one long request, several concurrent requests, vision input, or higher precision: those requirements spend memory differently. A successful short conversation is sufficient to prove that the model loads, but it cannot settle the hardware decision for a sustained long-context workload.

If the unresolved question is whether this model is the right choice in the first place, use the separate [Qwen3.8 Flash, Next, and 27B comparison](https://blog.laozhang.ai/en/posts/qwen3-8-flash-next-vs-qwen3-8-27b). Once you have chosen 27B, keep the exact checkpoint, runtime version, cache precision, input length, output allowance, and concurrency together as your deployment specification.

## Sources

External pages this guide links to, in the order they appear. Last updated 2026-09-06.

- [official configuration](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json) (huggingface.co)
- [RTX 5090 has 32GB of GDDR7 and no NVLink](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/) (nvidia.com)
- [official weight index](https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/model.safetensors.index.json) (huggingface.co)
- [official safetensors files](https://huggingface.co/Qwen/Qwen3.8-27B-FP8/tree/main) (huggingface.co)
- [Unsloth revision 4ca720788d1e01f1bff70c033e0d0028fd02e502](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/tree/4ca720788d1e01f1bff70c033e0d0028fd02e502) (huggingface.co)
- [llama.cpp block definitions](https://github.com/ggml-org/llama.cpp/blob/master/ggml/src/ggml-common.h) (github.com)
- [SGLang cookbook](https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B) (docs.sglang.io)
- [llama-server documentation](https://github.com/ggml-org/llama.cpp/tree/master/tools/server) (github.com)
- [Qwen3.8-27B recipe](https://recipes.vllm.ai/Qwen/Qwen3.8-27B) (recipes.vllm.ai)
- [MiaAI-Lab single-5090 configuration](https://github.com/MiaAI-Lab/Qwen3.8-27B-NVFP4-RTX-5090) (github.com)