AIFIRST Running Qwen3.8-27B-Q4 at Max Context on 32 GB GPU Without KV Cache Quantization
The brief
A detailed guide describes running Qwen3.8-27B at Q4 quantization on a 32 GB GPU while keeping KV cache at full precision, which the author argues produces measurably better output than KV quantization.
Key points
- The approach requires careful layer offloading and attention window management.
- The post includes specific configuration parameters for reproducibility.
Sources
- redditreddit.com