Model Downloads (GGUF)
One table per artifact: the model itself plus the companion files — projector, VAE, text encoder, second expert, speculative drafter — that each family needs alongside it.
Model downloads (GGUF)
| Architecture | Model | Download | Companion files (mmproj / MTP draft) |
|---|---|---|---|
| DeepSeek V4 | DeepSeek-V4-Flash-0731 (284B MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF | One subdirectory per quant (UD-Q8_K_XL/, UD-IQ4_XS/, UD-IQ1_S/, …), each a multi-shard set — point --model at the -00001-of- shard. Text only; weights layer-split across every visible GPU |
| DeepSeek V4 | DSpark speculative drafter (optional) | bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF | DSpark-drafter-Q2K-Q8-0731.gguf (7.0 GB) for the 0731 release, loaded with --draft-model for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see MODEL_DOWNLOADS.md. Drafters for other architectures are a different design and are not supported |
| GLM 5.x | GLM-5.2 (744B-A40B MoE) | unsloth/GLM-5.2-GGUF | One subdirectory per quant (UD-Q4_K_XL/, UD-IQ2_XXS/, …), each a multi-shard set — point --model at the -00001-of- shard and GgufFile reads the rest. Text only; no projector or drafter |
| Gemma 4 | gemma-4-E4B-it | ggml-org/gemma-4-E4B-it-GGUF | Recommended public artifact for the verified E4B Q8_0 native-GGML tier: gemma-4-E4B-it-Q8_0.gguf. Modalities: mmproj-gemma-4-E4B-it-Q8_0.gguf (same repo). MTP draft: AtomicChat/gemma-4-E4B-it-assistant-GGUF |
| Gemma 4 | gemma-4-12B-it (QAT) | unsloth/gemma-4-12B-it-qat-GGUF | mmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-12B-it.gguf (both in the same repo) |
| Gemma 4 | gemma-4-31B-it | ggml-org/gemma-4-31B-it-GGUF | mmproj: mmproj-gemma-4-31B-it-Q8_0.gguf (same repo) |
| Gemma 4 | gemma-4-26B-A4B-it (MoE) | ggml-org/gemma-4-26B-A4B-it-GGUF | mmproj: mmproj-gemma-4-26B-A4B-it-Q8_0.gguf (same repo) |
| Gemma 4 | gemma-4-26B-A4B-it (MoE, QAT) | unsloth/gemma-4-26B-A4B-it-qat-GGUF | mmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-26B-A4B-it.gguf (same repo). Alt draft: AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — pair drafts only with the matching target size |
| Gemma 3 | gemma-3-4b-it (QAT, official) | google/gemma-3-4b-it-qat-q4_0-gguf | mmproj: mmproj-model-f16-4B.gguf (same repo). ⚠️ Gated: downloads require a Hugging Face login + accepting Google's Gemma license |
| Gemma 3 | gemma-3-4b-it (non-gated alternative) | ggml-org/gemma-3-4b-it-GGUF | mmproj: mmproj-model-f16.gguf (same repo) |
| Qwen 3 | Qwen3-4B | Qwen/Qwen3-4B-GGUF | — (text only) |
| Qwen 3.5 / 3.6 | Qwen3.5-9B | unsloth/Qwen3.5-9B-GGUF | mmproj: mmproj-F16.gguf (same repo) |
| Qwen 3.5 / 3.6 | Qwen3.5-35B-A3B (MoE) | ggml-org/Qwen3.5-35B-A3B-GGUF | mmproj: mmproj-Qwen3.5-35B-A3B-Q8_0.gguf (same repo) |
| Qwen 3.5 / 3.6 | Qwen3.6-35B-A3B (MoE, NextN MTP) | unsloth/Qwen3.6-35B-A3B-MTP-GGUF | mmproj: mmproj-F16.gguf (same repo). These GGUFs retain the embedded NextN block for --mtp-spec; the base-repo GGUFs (unsloth/Qwen3.6-35B-A3B-GGUF) strip it and silently fall back to standard decode |
| GPT OSS | gpt-oss-20b (MoE) | ggml-org/gpt-oss-20b-GGUF | — (text only) |
| Nemotron-H | Nemotron-H-8B-Reasoning-128K | bartowski/nvidia_Nemotron-H-8B-… | — (text only) |
| Nemotron-H | Nemotron-H-47B-Reasoning-128K | bartowski/nvidia_Nemotron-H-47B-… | — (text only) |
| Nemotron-H | Nemotron 3 Nano Omni 30B-A3B (image) | unsloth/NVIDIA-Nemotron-3-Nano-Omni-… | mmproj: mmproj-BF16.gguf (same repo) — required for image input. Audio is not functional: it needs a Parakeet audio mmproj that this distribution does not ship |
| Mistral 3 | Mistral-Small-3.1-24B-Instruct | bartowski/mistralai_Mistral-Small-3.1-24B-… | Pixtral mmproj: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf (same repo) |
| DiffusionGemma | diffusiongemma-26B-A4B-it | unsloth/diffusiongemma-26B-A4B-it-GGUF | — |
| Muse-Glimmer | Muse-Glimmer-30B | unsloth/Muse-Glimmer-30B-GGUF | mmproj: mmproj-Muse-Glimmer-30B-Q8_0.gguf (same repo) for image input |
| Muse-Glimmer | DFlash speculative drafter (optional) | unsloth/Muse-Glimmer-30B-GGUF | dflash-*.gguf — a 5-layer block drafter loaded with --draft-model. Verification is greedy against the target, so the emitted stream is the plain-greedy stream |
| Qwen-Image-Edit | MMDiT DiT (the --model GGUF) | unsloth/Qwen-Image-Edit-2511-GGUF | e.g. qwen-image-edit-2511-Q4_K_M.gguf |
| Qwen-Image-Edit | Qwen-Image VAE (required) | QuantStack/Qwen-Image-Edit-GGUF | VAE/Qwen_Image-VAE.safetensors — place next to the DiT or pass --qwen-image-vae |
| Qwen-Image-Edit | Qwen2.5-VL-7B text encoder (required) | unsloth/Qwen2.5-VL-7B-Instruct-GGUF | Optional vision mmproj: mmproj-BF16.gguf (same repo) for image-grounded edits |
| Qwen-Image-Edit | Lightning LoRA (optional, 4/8-step) | lightx2v/Qwen-Image-Edit-2511-Lightning | Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors via --qwen-image-lora |
| Wan video | Step-distilled Wan DiT — the fast lane (recommended) | hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF | Wan2_2-TI2V-5B-Turbo-Q8_0.gguf (5.40 GB) — note the underscore in Wan2_2; smaller tiers down to Q2_K (1.86 GB). Detected from the file name, so no flag changes: 4 DiT passes instead of 100, guidance off. For A14B image-to-video use jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF and take both the high_noise/ and low_noise/ files. Neither repo ships a VAE or a text encoder — take those from the two rows below |
| Wan video | Base Wan DiT (full step recipe) | QuantStack/Wan2.2-TI2V-5B-GGUF | Wan2.2-TI2V-5B-Q8_0.gguf (5.40 GB), with VAE/Wan2.2_VAE.safetensors bundled in the same repo. Also Wan2.2-I2V-A14B and Wan2.2-T2V-A14B (download both the HighNoise and LowNoise experts; both repos bundle the Wan 2.1 VAE), Wan2.1-T2V-14B and Wan2.1-T2V-1.3B |
| Wan video | UMT5-XXL text encoder (required) | city96/umt5-xxl-encoder-gguf | umt5-xxl-encoder-Q8_0.gguf (6.04 GB; Q5_K_M 4.15 GB and Q4_K_M 3.66 GB for tighter memory) — place next to the DiT or pass --wan-te |
| Wan video | video VAE (required) | wan_2.1_vae.safetensors | The DiT decides which one it needs, not you: Wan 2.1 and A14B use wan_2.1_vae.safetensors, TI2V-5B uses Wan2.2_VAE.safetensors. Both QuantStack A14B repos and the TI2V-5B repo already bundle theirs under VAE/, so this separate download is only needed for the Wan 2.1 checkpoints and the distilled repos. Place it next to the DiT (a VAE/ subfolder works) or pass --wan-vae |
🧩
Multimodal models need a projector (mmproj) file. Download the matching file and pass its exact path with --mmproj. The server never auto-detects it, and CLI auto-detection recognizes only a small set of legacy filenames.