Qwen3.8-27B VRAM Guide: RTX 5090, Dual GPUs, and 262K Context

For a single RTX 5090, start with quantized weights, one request at a time, and a 32,768-token context. Treat 262,144 tokens as a separate configuration to validate. Four-bit weights make the model much smaller, but they do not automatically make its attention cache four-bit. That distinction can change a full-context memory estimate by more than 10 GiB.

Qwen3.8-27B has a native maximum context of 262,144 tokens, shared by the input and generated output. It is a dense vision-language model with a hybrid attention architecture, so both conventional “parameters times two” estimates and generic Transformer KV-cache calculators miss important details. The model's official configuration supplies the dimensions needed for a more useful estimate.

The practical choice is between a straightforward GGUF deployment with explicit cache settings, a checkpoint-specific NVFP4 serving setup, and two GPUs with a runtime that distributes the model. Below, file sizes and calculations are separated from configurations reported by runtime maintainers and independent users. None of those reports establishes performance on every machine with the same GPU name.

Start with the weights you will actually load

The desktop RTX 5090 has 32GB of GDDR7 and no NVLink. Check the capacity and available memory reported by your driver before making a budget; a GPU running a display may already have allocations.

bash
nvidia-smi --query-gpu=index,name,memory.total,memory.used,memory.free --format=csv

Keep units consistent. Decimal GB means 1,000,000,000 bytes; GiB means 1,073,741,824 bytes. Convert the driver's reported units before subtracting the figures below. The marketed “32GB” designation should not replace the actual memory reading.

The BF16 figure is tensor storage from the official weight index, excluding small file headers. FP8 is the sum of the official safetensors files. GGUF figures refer to Unsloth revision 4ca720788d1e01f1bff70c033e0d0028fd02e502, checked September 6, 2026. These are storage measurements, not measured resident VRAM.

Two consequences matter when choosing a download. First, official FP8 is not exactly half the BF16 size: scales and components retained at other precisions affect the result. Second, a label such as “Q4” does not identify one universal checkpoint. Publisher, quantization scheme, mixed precision choices, and included components matter. Do not substitute an unrelated four-bit file into a memory or quality claim about UD-Q4_K_M.

GGUF and NVFP4 also require compatible loading paths. A vLLM NVFP4 recipe is not a set of switches that can be applied to an arbitrary GGUF file. Choose the runtime and supported checkpoint together, then adjust context and cache precision within that combination.

Choose weight precision with a small set of your real tasks. If a four-bit checkpoint gives acceptable answers, spend the remaining memory on the context and concurrency you need. If it fails important tasks, compare a higher-precision checkpoint while keeping the cache settings and inputs fixed. File size alone cannot rank answer quality, and changing both weight and cache precision at once makes it harder to identify the cause of a regression.

Why the KV cache changes the answer

Qwen3.8-27B has 64 layers, but only 16 use full attention; the other 48 use linear attention. Each full-attention layer has four KV heads with a head dimension of 256. Applying the ordinary growing KV-cache formula to all 64 layers would overestimate this part of memory fourfold. The linear-attention layers still need recurrent state; they do not cost zero memory. These dimensions come from the model configuration.

For one sequence, the full-attention KV payload at F16 or BF16 is:

text
2 × 16 × 4 × 256 × 2 = 65,536 bytes per stored token
│   │    │    │    └─ bytes per value
│   │    │    └────── head dimension
│   │    └─────────── KV heads
│   └──────────────── full-attention layers
└──────────────────── K and V

That is 64 KiB per token, or 16 GiB at the native maximum. This arithmetic covers the full-attention payload only, assuming both K and V use the stated precision.

FP8 and q8_0 are not interchangeable labels. The llama.cpp block definitions allocate 34 bytes for 32 values in q8_0 and 18 bytes for 32 values in q4_0. Their scales explain why the payload is slightly larger than ideal eight-bit or four-bit storage. Allocator overhead and runtime-specific padding come afterward.

This gives a revealing single-card example. With Unsloth UD-Q4_K_M and 262,144 stored tokens:

text
F16 KV:   15.334 + 16.0 = 31.334 GiB
q8_0 KV:  15.334 +  8.5 = 23.834 GiB
q4_0 KV:  15.334 +  4.5 = 19.834 GiB

The F16 case leaves too little room for ordinary runtime needs on a 32GB RTX 5090. The quantized-cache cases have more plausible headroom, but neither is a fit guarantee: the first term is still a weight-file estimate, and the sums omit recurrent state, activations, workspaces, CUDA graphs, and optional features. Quantized KV also needs task-quality testing; a successful allocation does not establish that cache compression preserves the answers you need.

Budget for the peak, including optional features

A more complete planning equation is:

text
Peak VRAM = resident weights
          + full-attention KV cache
          + linear-attention recurrent state
          + activations, workspaces, graphs, and allocator overhead
          + optional vision and speculative-decoding allocations

The SGLang cookbook reports approximately 153.9 MB for one GDN state slot in FP32 or 78.4 MB in BF16. The runtime's reserved slot count and speculative intermediates determine the total. A fixed-size state per slot is helpful for long-context scaling, but it remains an allocation you must account for.

For GGUF vision inference, the matching mmproj-F16.gguf in the same Unsloth snapshot adds 927,607,488 bytes, about 0.864 GiB, of projector weights. Image-processing buffers add further memory. A text-only fit therefore does not establish a multimodal fit.

Concurrency changes the budget again. Two unrelated sequences with 131,072 cached tokens each require the same full-attention payload as one sequence with 262,144 tokens, before their additional per-sequence costs. Prefix sharing may help when a runtime actually reuses shared tokens, but it should not be assumed for unrelated requests. Multi-token prediction (MTP), used for speculative decoding, adds its own allocations and compatibility requirements; enable it after the basic workload is stable.

A practical first configuration for one RTX 5090

For a text-only GGUF deployment, begin with UD-Q4_K_M, a 32,768-token context, and one active sequence. At this length the calculated weight-file size plus q8_0 full-attention cache is about 16.397 GiB, leaving substantially more room to discover actual runtime overhead than a full-context first launch.

The following starting configuration adapts flags from the llama-server documentation. It requires validation on your hardware and installed build. Replace the model path with your downloaded file and check your server's help for model support and flag availability.

bash
llama-server --version
llama-server --help

llama-server \
  --model /path/to/Qwen3.8-27B-UD-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 32768 \
  --parallel 1 \
  --flash-attn on \
  --cache-type-k q8_0 \
  --cache-type-v q8_0

The high GPU-layer request is intended to offload all available model layers; confirm what actually loaded from the startup log. Keep the context at 32,768 until a representative request completes. Then increase it to 65,536, 131,072, and finally 262,144, repeating the long-input checks below. Changing one variable at a time makes a failure easier to attribute to cache capacity, prefill memory, or a particular quantization setting.

If you prefer vLLM, its Qwen3.8-27B recipe includes a single-RTX-5090 Inferact NVFP4 example with tensor parallelism set to one, --max-model-len 32768, FP8 KV, --enforce-eager, and --reasoning-parser qwen3. The maintainers report a graph-capture out-of-memory failure without eager execution in that tested setup. Use the checkpoint and version stated in the recipe; eager execution is a documented response to that failure, not a universal requirement for every NVFP4 deployment.

What the single-card 262K report actually establishes

The MiaAI-Lab single-5090 configuration reports a 262,144-token setting using RadixArk NVFP4, vllm==0.27.1 plus a backport of change #40914, turboquant_4bit_nc cache, a 5.5 GiB KV pool, one sequence, a 512-token batch limit, and MTP3.

That is a specific reproducibility lead for users willing to maintain the stated software combination. It is not evidence that a stock vLLM install, another four-bit checkpoint, or the 32K baseline above will reach the same capacity unchanged. The repository also describes garbled output with the stock path and crashes involving MTP with concurrency. Check those constraints before adopting it as a service configuration.

A reported maximum context and a reported decode rate must also describe the same workload before they can support a full-context performance claim. A speed figure from a shorter request cannot tell you how long a 262K prefill will take or what decode performance follows it.

When a second GPU is the better tradeoff

Two GPUs become useful when you want less aggressive weight quantization, higher cache precision, more simultaneous requests, or enough margin to avoid a highly specialized single-card setup. They do not create one automatically pooled memory space, and they do not imply twice the speed.

At full context, Q8_0 weights plus F16 full-attention KV total approximately 43.052 GiB before other allocations. That exceeds one 32GB card but is a plausible aggregate starting budget for two such cards. Original BF16 weights plus the same cache total approximately 67.75 GiB, already exceeding two nominal 32GB cards before overhead. A second card therefore does not remove the need to choose precision deliberately.

Two 24GB cards may appear adequate for the 43.052 GiB subtotal, but aggregate arithmetic cannot establish that each GPU's assigned layers, cache, buffers, and free memory fit. An uneven split or a larger allocation on the main GPU can still fail.

Choose how the runtime distributes work

For GGUF, llama.cpp documents --split-mode layer as splitting layers and KV across GPUs. You can extend the single-card command with:

bash
--split-mode layer --tensor-split 1,1

The proportions request an equal distribution; they do not certify equal peak allocation. Check both GPUs while processing an actual prompt. For cards with different free memory, adjust the proportions to leave room for each device's other allocations. The documented row mode handles KV and intermediate results differently, with these on the main GPU, so do not reuse a layer-split budget without checking the runtime's split-mode behavior.

For vLLM, the upstream recipe uses tensor parallelism of two on two RTX 5090s, distributing computation within the model. Its tested build is 0.26.1rc1.dev608+g99a10304d; the broad minimum-version heading is not the same as that exact tested environment. The reported startup results are:

These token counts describe the startup cache pool, which can serve token storage across requests. They do not raise the model's native per-request context limit, and they are not proof of a successful prefill at every length represented by the pool. The weights-per-GPU measurements also belong to this runtime and checkpoint combination; they should not be substituted for GGUF residency estimates.

Because the RTX 5090 has no NVLink, inspect your PCIe arrangement before choosing tensor parallelism for performance. Use nvidia-smi topo -m to see the reported topology, then benchmark your actual workload. Capacity alone cannot predict the communication cost or whether layer splitting and tensor parallelism will behave similarly on your motherboard.

CPU offload is another capacity option when the available GPU memory is insufficient. It changes where memory must be available and may move work or transfers off the GPU; it does not make a partly offloaded result comparable to a fully resident GPU benchmark. Record offload alongside the checkpoint and context when comparing configurations.

Verify that the requested context is usable

“262K” is a shorthand for the model's 262,144-token limit; “256K” in binary counting can refer to that same number. The number you set at launch is only the beginning of validation. Distinguish these four observations:

1. Configured: the server accepts the requested maximum context setting.
2. Allocated: its startup log reports sufficient cache capacity for the intended requests.
3. Processed: a tokenized long input completes prefill and generation without truncation or an out-of-memory error.
4. Useful: the output correctly uses relevant information throughout that input.

Reserve output space inside the limit. For example, a total input of 258,048 tokens leaves 4,096 tokens under 262,144. That input count must already include system instructions, chat-template tokens, tool definitions, and any other material passed to the model. The file's word count or character count is not a tokenizer measurement.

Create a test document with distinct, verifiable facts near its beginning, middle, and end. Ask questions that require those facts, plus at least one answer that depends on combining distant sections. Record the actual token count, whether truncation is enabled, time to first token, generation rate, and peak memory on every GPU. Run this with the output allowance and concurrency you intend to use, rather than reducing them only to pass the test.

Compare the resulting answers against a shorter context or a higher-precision configuration using the same facts. This helps distinguish “the server stayed alive” from “the compressed cache still supports the task.” Add images and MTP separately if you need them, and repeat the representative workload after each addition.

The distinction matters even when following official runtime documentation. The SGLang consumer configurations were validated with an 8,192-token input, a 1,024-token output, and concurrency one. Those conditions are useful reproducibility information; they are not a 262K workload test.

Diagnose failures by when they occur

For an existing single RTX 5090, the most informative next step is a conservative quantized launch followed by measured context increases. For a hardware purchase, first decide whether your requirement is one long request, several concurrent requests, vision input, or higher precision: those requirements spend memory differently. A successful short conversation is sufficient to prove that the model loads, but it cannot settle the hardware decision for a sustained long-context workload.

If the unresolved question is whether this model is the right choice in the first place, use the separate Qwen3.8 Flash, Next, and 27B comparison. Once you have chosen 27B, keep the exact checkpoint, runtime version, cache precision, input length, output allowance, and concurrency together as your deployment specification.