Model Downloads (GGUF)
One table per artifact: the model itself plus the companion files — projector, VAE, text encoder, second expert, speculative drafter — that each family needs alongside it.
Sentence encoders: Snowflake Arctic Embed L v2.0 / all-MiniLM-L6-v2, served through --embeddings and OpenAI/Ollama embedding APIs. Embedding models, supported backends, and validation →
Model downloads (GGUF)
| Architecture | Model | Download | Companion files (mmproj / MTP draft) |
|---|---|---|---|
Embedding encoder (bert / XLM-R) | Snowflake Arctic Embed L v2.0, Q8_0 (1024 dimensions) | fisher046/snowflake-arctic-embed-l-v2.0-Q8_0-GGUF | snowflake-arctic-embed-l-v2.0-q8_0.gguf, about 635 MB; serve with --embeddings. Pinned revisions and checksums: embedding guide |
Embedding encoder (bert) | all-MiniLM-L6-v2, Q8_0 (384 dimensions) | second-state/All-MiniLM-L6-v2-Embedding-GGUF | all-MiniLM-L6-v2-Q8_0.gguf, about 25 MB; serve with --embeddings |
| DeepSeek V4.1 | DeepSeek-V4.1-Flash (deepseek41) | smalinin/DeepSeek-V4.1-Flash-GGUF @ d1de55c | Repaired Q2_K-Q5/ set: ten shards, 312.349 GiB; keep all files and point --model to DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf. F32 mHC matrices, BF16 Engram gates and Q5_K Engram tables; no separate Engram file. Optional eng/dsv41-prepare-vision.py supplies image/video conditioning. ggml_cuda is the serving path; local layer split or routed-MoE TP (degrees 2–8), with explicit memory/offload constraints. Historical vcruz305 seven-shard results do not qualify this repair. Recipe and validation scope → |
| DeepSeek V4 | DeepSeek-V4-Flash-0731 (284B MoE) | unsloth/DeepSeek-V4-Flash-0731-GGUF | One subdirectory per quant (UD-Q8_K_XL/, UD-IQ4_XS/, UD-IQ1_S/, …), each a multi-shard set — point --model at the -00001-of- shard. Text only; --layer-split N places whole layers across N local GPUs (default: one device) |
| DeepSeek V4 | DSpark speculative drafter (optional) | bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF | DSpark-drafter-Q2K-Q8-0731.gguf (7.0 GB) for the 0731 release, loaded with --draft-model for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see MODEL_DOWNLOADS.md. Drafters for other architectures are a different design and are not supported |
| GLM 5.x | GLM-5.2 (744B-A40B MoE) | unsloth/GLM-5.2-GGUF | One subdirectory per quant (UD-Q4_K_XL/, UD-IQ2_XXS/, …), each a multi-shard set — point --model at the -00001-of- shard and GgufFile reads the rest. Text only; no projector or drafter |
| GLM 5.x | GLM-5.3 (256 routed experts, glm-dsa) | unsloth/GLM-5.3-GGUF | One subdirectory per quant, each a multi-shard set — point --model at the -00001-of- shard. Text only; no mmproj is published. It carries the NextN block, but that block ships no LM head of its own, so --spec engages on single-device or explicit --layer-split N placement, with no active tensor parallelism |
| GLM 5.x | GLM-5.3-Flash (320B MoE, glm5next) | unsloth/GLM-5.3-Flash-GGUF | One subdirectory per quant, each a multi-shard set — point --model at the -00001-of- shard. mmproj: mmproj-BF16.gguf (same repo, the GLM-OCR ViT) for image input. Loads through the same native executor as GLM-5.2; --layer-split N selects whole-layer placement across N local GPUs, while --tp N on a GGML GPU backend selects native local/single-process tensor parallelism |
| Gemma 4 | gemma-4-E4B-it | ggml-org/gemma-4-E4B-it-GGUF | Recommended public artifact for the verified E4B Q8_0 native-GGML tier: gemma-4-E4B-it-Q8_0.gguf. Modalities: mmproj-gemma-4-E4B-it-Q8_0.gguf (same repo). MTP draft: AtomicChat/gemma-4-E4B-it-assistant-GGUF |
| Gemma 4 | gemma-4-12B-it (QAT) | unsloth/gemma-4-12B-it-qat-GGUF | mmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-12B-it.gguf (both in the same repo) |
| Gemma 4 | gemma-4-31B-it | ggml-org/gemma-4-31B-it-GGUF | mmproj: mmproj-gemma-4-31B-it-Q8_0.gguf (same repo) |
| Gemma 4 | gemma-4-26B-A4B-it (MoE) | ggml-org/gemma-4-26B-A4B-it-GGUF | mmproj: mmproj-gemma-4-26B-A4B-it-Q8_0.gguf (same repo) |
| Gemma 4 | gemma-4-26B-A4B-it (MoE, QAT) | unsloth/gemma-4-26B-A4B-it-qat-GGUF | mmproj: mmproj-BF16.gguf; MTP draft: mtp-gemma-4-26B-A4B-it.gguf (same repo). Alt draft: AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — pair drafts only with the matching target size |
| Qwen 3.5 / 3.6 | Qwen3.5-9B | unsloth/Qwen3.5-9B-GGUF | mmproj: mmproj-F16.gguf (same repo) |
| Qwen 3.5 / 3.6 | Qwen3.5-35B-A3B (MoE) | ggml-org/Qwen3.5-35B-A3B-GGUF | mmproj: mmproj-Qwen3.5-35B-A3B-Q8_0.gguf (same repo) |
| Qwen 3.5 / 3.6 | Qwen3.6-35B-A3B (MoE, NextN MTP) | unsloth/Qwen3.6-35B-A3B-MTP-GGUF | UD-Q4_K_M is about 21.11 GiB. mmproj: mmproj-F16.gguf (same repo). These GGUFs retain the embedded NextN block for --spec; the base-repo GGUFs (unsloth/Qwen3.6-35B-A3B-GGUF) strip it and silently fall back to standard decode |
| Qwen 3.5 / 3.6 | Qwen3.8-27B (dense hybrid, embedded NextN) | unsloth/Qwen3.8-27B-GGUF | e.g. Qwen3.8-27B-UD-Q4_K_XL.gguf, the file config/agent-qwen3.8-27b.json pins by commit and SHA-256; it keeps the embedded NextN block for --spec. mmproj: mmproj-BF16.gguf (same repo). Loads through the same qwen35 plug-in as Qwen 3.5 (details) |
| Qwen 3.5 / 3.6 | Qwen3.8-27B DFlash2 block drafter (optional) | z-lab/Qwen3.8-27B-DFlash2-GGUF | Loaded with --draft-model; it replaces the embedded NextN block when attached. What it gains depends on the workload — see Qwen 3.5 / 3.6 |
| Qwen 3.8 Flash Next | Qwen3.8-Flash-Next (hybrid MoE, qwen4exp) | unsloth/Qwen3.8-Flash-Next-GGUF | One subdirectory per quant, each a multi-shard set — point --model at the -00001-of- shard. On a 48 GB Mac, ggml_metal runs UD-Q2_K_XL (78.9 GB in three shards) larger than memory, reading its n-gram table and offloaded experts from the SSD on demand; TensorAgent offers it there (details). mmproj: mmproj-BF16.gguf (same repo) for image and video input. MTP draft (optional): the shared head mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (2,786,568,256 bytes at this repo's revision 38bb39ee), loaded with --draft-model on a GGML backend — it pays on copy-heavy output and slows prose. --layer-split N here is a layer split across N GPUs, not tensor parallelism |
| GPT OSS | gpt-oss-20b (MoE) | ggml-org/gpt-oss-20b-GGUF | — (text only) |
| Nemotron-H | Nemotron-H-8B-Reasoning-128K | bartowski/nvidia_Nemotron-H-8B-… | — (text only) |
| Nemotron-H | Nemotron-H-47B-Reasoning-128K | bartowski/nvidia_Nemotron-H-47B-… | — (text only) |
| Nemotron-H | Nemotron 3 Nano Omni 30B-A3B (image) | unsloth/NVIDIA-Nemotron-3-Nano-Omni-… | mmproj: mmproj-BF16.gguf (same repo) — required for image input. It carries no audio tower: audio needs a separately prepared companion GGUF with the Parakeet tower (conversion scripts in eng/nemotron-audio/; no public repository ships one), and is refused without it |
| Nemotron 3.5 | Nemotron-3.5-Lightning-30B-A3B (23 Mamba-2 + 23 MoE + 6 attention) | unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF | No drafter: Nemotron-H refuses every speculator, because its multi-token verify and single-token decode use different kernels, so NVIDIA's DSpark drafter for this model is not attached (the server refuses to start with it) |
| Mistral 3 | Mistral-Small-3.1-24B-Instruct | bartowski/mistralai_Mistral-Small-3.1-24B-… | Q4_K_M is about 13.35 GiB. Pixtral mmproj: mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf (same repo) |
| Hunyuan Dense | Tencent dense Hunyuan (hunyuan-dense), e.g. Hy-MT2 | Any GGUF declaring that architecture | Selected from general.architecture, not the file name. Text only, single device, no projector and no drafter |
| Bonsai2 | Ternary-Bonsai-2-27B PQ2_0 / PTQ1_0 (qwen35 + prism.hadamard.*) | prism-ml/Ternary-Bonsai-2-27B-gguf @ b072e1d | Ternary-Bonsai-2-27B-PQ2_0.gguf / -PTQ1_0.gguf, exact bytes and SHA-256 in the Bonsai2 card; Apache-2.0. mmproj: Ternary-Bonsai-2-27B-mmproj-BF16.gguf or -mmproj-Q8_0.gguf (same repo) for image input. Single-device GGML backends only |
| DiffusionGemma | diffusiongemma-26B-A4B-it | unsloth/diffusiongemma-26B-A4B-it-GGUF | Q4_K_M is about 15.65 GiB. Vision tower for image input (optional): no GGUF carries it, so load model-00011-of-00011.safetensors (2.84 GB) from google/diffusiongemma-26B-A4B-it with --mmproj; config/diffusiongemma-26b-a4b-q4.json, -q3.json and jev-diffusiongemma-q4.json fetch it. No audio tower exists for this checkpoint |
| Muse-Glimmer | Muse-Glimmer-30B | unsloth/Muse-Glimmer-30B-GGUF | mmproj: mmproj-Muse-Glimmer-30B-Q8_0.gguf (same repo) for image input |
| Muse-Glimmer | DFlash speculative drafter (optional) | unsloth/Muse-Glimmer-30B-GGUF | dflash-*.gguf — a 5-layer block drafter loaded with --draft-model. Verification is greedy against the target, so the emitted stream is the plain-greedy stream |
| Qwen-Image-2.1 | Diffusion transformer (the --model GGUF) | Abiray/Qwen-Image-2.1-GGUF | qwen_image_2.1_Q4_K_M.gguf. config/qwen-image-2.1.json downloads it and the three companions below with pinned revisions and SHA-256 checksums (about 10.29 GiB in total). Metadata-free 2.1 GGUFs, such as Unsloth's qwen-image-2.1-Q8_0.gguf, also load: the architecture is recognized from the tensor layout. Earlier Qwen-Image and Qwen-Image-Edit-2511 transformers are refused |
| Qwen-Image-2.1 | Dedicated 2.1 VAE (required) | Comfy-Org/Qwen-Image-2.1 | vae/qwen_image_2.1_vae_bf16.safetensors — place next to the DiT or pass --qwen-image-vae |
| Qwen-Image-2.1 | Qwen3-VL-8B text encoder (required) | Qwen/Qwen3-VL-8B-Instruct-GGUF | Qwen3VL-8B-Instruct-Q4_K_M.gguf — place next to the DiT or pass --qwen-image-vl |
| Qwen-Image-2.1 | Vision encoder for editing | Qwen/Qwen3-VL-8B-Instruct-GGUF | mmproj-Qwen3VL-8B-Instruct-F16.gguf — place next to the DiT or pass --qwen-image-mmproj. The auto-scan matches only names containing qwen3vl-8b / qwen3-vl-8b, so a projector saved as mmproj-BF16.gguf (Unsloth's name) needs the flag |
| Qwen-Image-2.1-Turbo | Diffusion transformer, 8-step distillation (the --model GGUF) | AtomicChat/Qwen-Image-2.1-Turbo-GGUF | Qwen-Image-2.1-Turbo-AD-Q4_K.gguf (4.20 GB), or Qwen-Image-2.1-Turbo-Q8_0.gguf (7.59 GB), the transformer closest to full precision (with the same Q4_K_M text encoder). config/qwen-image-2.1-turbo.json downloads AD-Q4_K with a pinned revision and SHA-256, shares the 2.1 VAE, text encoder and mmproj above, and declares the checkpoint (--qwen-image-variant turbo) — see Qwen-Image-2.1-Turbo |
| Qwen-Image-2.1 | LoRA plug-ins (optional) | config/lora/*.json | Fourteen plug-ins loaded with --lora; each downloads its weights with a pinned revision and SHA-256 on first use. Step-distillation adapters and style / editing LoRAs — see LoRA plug-ins |
| MiniMax-H3 audio+video | FL2VA denoiser (the --model GGUF) — text and keyframes | unsloth/MiniMax-H3-GGUF | minimax_h3_fl2va_pruned-Q4_K.gguf (10.64 GiB) — text-to-video, image-to-video (the image is the first frame) and first-and-last-frame. H3 is CFG-distilled: --cfg 1.0 is required, and 4–8 steps is the fast operating point against a 20-step default. The published GGUFs carry no metadata at all, so TensorSharp identifies them by their tensors and reads the partition off the file name — keep fl2va in it if you rename or requantize |
| MiniMax-H3 audio+video | Ref2VA denoiser (the other --model GGUF) — text and references | unsloth/MiniMax-H3-GGUF | minimax_h3_ref2va_pruned-Q4_K.gguf (10.60 GiB) — the ref mode: up to nine identity and appearance references, in any mix of stills, clips and soundtracks (--ref-image, --ref-video, --ref-video-audio, --ref-audio), composing a new scene rather than frames the clip has to reproduce. Two checkpoints, not two settings: asking one for the other's mode fails with a message naming the file to load instead, and keyframes together with named references are refused outright. Keep ref2va in the file name — that is how the partition is read. The three rows below are shared with FL2VA, so adding this checkpoint later costs only its own 10.60 GiB |
| MiniMax-H3 audio+video | Qwen3-VL-32B text encoder (required) | unsloth/MiniMax-H3-GGUF | qwen3vl_32b_minimax_h3-Q4_K_M.gguf (16.97 GiB), or -Q2_K_M.gguf (12.20 GiB) to pair with a smaller denoiser. Truncated to 50 layers with the final norm removed, and freed before the denoise starts. It ships no tokenizer — see the callout below. Place next to the denoiser or pass --video-text-encoder |
| MiniMax-H3 audio+video | video VAE (required) | Comfy-Org/MiniMax-H3 | minimax_h3_video_vae_fp16.safetensors (5.21 GB) — 16× spatial / 4× temporal, with a pure-transformer decoder and no deconvolutions. Also mirrored under vae/ in the unsloth repo above. Place next to the denoiser or pass --video-vae |
| MiniMax-H3 audio+video | audio VAE (optional) | Comfy-Org/MiniMax-H3 | minimax_h3_audio_vae_fp32.safetensors (0.61 GB) — an alias-free BigVGAN that decodes the jointly generated audio latent to 32 kHz stereo, written as a sidecar .wav beside the MP4. Omit it and you still get video, just silent. Pass --audio-vae. On Ref2VA its encoder is also what a --ref-audio goes through, so dropping it costs reference soundtracks too |
| Wan video | Step-distilled Wan DiT — the fast lane (recommended) | hum-ma/Wan2.2-TI2V-5B-Turbo-GGUF | Wan2_2-TI2V-5B-Turbo-Q8_0.gguf (5.40 GB) — note the underscore in Wan2_2; smaller tiers down to Q2_K (1.86 GB). Detected from the file name, so no flag changes: 4 DiT passes instead of 100, guidance off. For A14B image-to-video use jayn7/WAN2.2-I2V_A14B-DISTILL-LIGHTX2V-4STEP-GGUF and take both the high_noise/ and low_noise/ files. Neither repo ships a VAE or a text encoder — take those from the two rows below |
| Wan video | Base Wan DiT (full step recipe) | QuantStack/Wan2.2-TI2V-5B-GGUF | Wan2.2-TI2V-5B-Q8_0.gguf (5.40 GB), with VAE/Wan2.2_VAE.safetensors bundled in the same repo. Also Wan2.2-I2V-A14B and Wan2.2-T2V-A14B (download both the HighNoise and LowNoise experts; both repos bundle the Wan 2.1 VAE), Wan2.1-T2V-14B and Wan2.1-T2V-1.3B |
| Wan video | UMT5-XXL text encoder (required) | city96/umt5-xxl-encoder-gguf | umt5-xxl-encoder-Q8_0.gguf (6.04 GB; Q5_K_M 4.15 GB and Q4_K_M 3.66 GB for tighter memory) — place next to the DiT or pass --video-text-encoder |
| Wan video | video VAE (required) | wan_2.1_vae.safetensors | The DiT decides which one it needs, not you: Wan 2.1 and A14B use wan_2.1_vae.safetensors, TI2V-5B uses Wan2.2_VAE.safetensors. Both QuantStack A14B repos and the TI2V-5B repo already bundle theirs under VAE/, so this separate download is only needed for the Wan 2.1 checkpoints and the distilled repos. Place it next to the DiT (a VAE/ subfolder works) or pass --video-vae |
🧩
Multimodal models need a projector (mmproj) file. Download the matching file and pass its exact path with --mmproj. The server never auto-detects a chat model's projector (Qwen-Image-2.1's editing projector is the exception: both hosts find it beside the DiT, see its row above). The CLI, given an image without --mmproj (or, for Gemma 4 and Nemotron, an audio clip or video), looks beside the model only for the file names that family declares — for example *mmproj*.gguf for Qwen 3.8 Flash Next and GLM 5.x, deepseek41.vision.gguf for DeepSeek V4.1, the two Bonsai2 projector names — and never for the DiffusionGemma vision shard.
🔤
The MiniMax-H3 text-encoder GGUF carries no tokenizer. Auto-download can only fill in options that are flags, and the tokenizer is not one, so no config file will fetch it for you. Take vocab.json and merges.txt from the processor/ folder of MiniMaxAI/MiniMax-H3 and put them beside the encoder, or point TS_VIDEO_TOKENIZER at the folder that holds them.
📦
config/minimax-h3-fl2va.json and config/minimax-h3-ref2va.json name all four networks, so the first run fetches them (~35.5 GB in total) and later runs reuse them. Only the model entry differs between the two files, so the text encoder and both VAEs download once and are shared — the second checkpoint costs one denoiser, not a second full set. Each network is also loaded and released in turn, so peak VRAM is the largest single network and not the sum of the four.