Quantization and choosing a quant

A model is billions of numbers. Quantization stores each number with fewer bits, so the file is smaller, loads faster and fits a cheaper GPU. The trade is a small loss in quality.

Common quants

A quick estimate: size in GB is about the number of parameters (in billions) times the bits, divided by 8.

How to choose

1. Start with the default Q4_K_M on an L4. For most chat and support use it is hard to tell apart from the original.
2. If answers feel careless with facts, numbers or code, try Q6_K or Q8_0.
3. If a bigger model in Q4_K_M fits the same GPU as a smaller model in Q8_0, the bigger model usually wins.
4. For the best throughput with many users at once, use a safetensors or FP8 repo on vLLM.

Pick a quant

Add it after the repo:

bitsrun deploy hf:unsloth/Qwen3-8B-GGUF:Q8_0

Or paste the link to one .gguf file from the repo's file list. Ollama names choose their quant by tag, for example qwen3:8b-q8_0 where the library lists it.

Switch later

You do not have to redeploy. Open Performance on the endpoint card and choose Fastest, Balanced or Best quality. Each shows its quality badge (Full, Near full, Reduced or Low quality), speed, GPU and price before you switch. See Performance.

What can go wrong