# Quantization and choosing a quant

> A quant is a smaller copy of a model; Q4_K_M is the default balance of size and quality, and higher numbers such as Q8_0 are closer to the original but need more memory.

Source: https://bitsrun.com/docs/models/quantization/
Updated: 2026-11-09

A model is billions of numbers. **Quantization** stores each number with fewer bits, so the file is smaller, loads faster and fits a cheaper GPU. The trade is a small loss in quality.

## Common quants

| Quant | Bits per number (about) | Size of an 8B model | Quality |
|---|---|---|---|
| `Q4_K_M` | 4.5 | about 5 GB | Good. The default. |
| `Q5_K_M` | 5.5 | about 6 GB | Better |
| `Q6_K` | 6.5 | about 7 GB | Very close to the original |
| `Q8_0` | 8.5 | about 9 GB | Near full quality |
| `F16` / `BF16` | 16 | about 16 GB | Full quality, the original |
| `FP8` (safetensors) | 8 | about 9 GB | Near full quality on vLLM |

A quick estimate: size in GB is about the number of parameters (in billions) times the bits, divided by 8.

## How to choose

1. Start with the default `Q4_K_M` on an L4. For most chat and support use it is hard to tell apart from the original.
2. If answers feel careless with facts, numbers or code, try `Q6_K` or `Q8_0`.
3. If a bigger model in `Q4_K_M` fits the same GPU as a smaller model in `Q8_0`, the bigger model usually wins.
4. For the best throughput with many users at once, use a safetensors or FP8 repo on vLLM.

## Pick a quant

Add it after the repo:

```bash
bitsrun deploy hf:unsloth/Qwen3-8B-GGUF:Q8_0
```

Or paste the link to one `.gguf` file from the repo's file list. Ollama names choose their quant by tag, for example `qwen3:8b-q8_0` where the library lists it.

## Switch later

You do not have to redeploy. Open **Performance** on the endpoint card and choose **Fastest**, **Balanced** or **Best quality**. Each shows its quality badge (Full, Near full, Reduced or Low quality), speed, GPU and price before you switch. See [Performance](https://bitsrun.com/docs/endpoints/performance/).

## What can go wrong

| Message | What to do |
|---|---|
| "This repo has no file with that quantization." | Leave it empty to use `Q4_K_M`, or name one the repo lists. |
| "This repo only has GGUF files split into parts" | Split files are not supported yet. Pick a repo with single-file quants. |
| "This repo keeps its model files in folders." | Open the `.gguf` file you want on Hugging Face and paste its link. |