Text & LLM Models

Copy-paste download and run recipes for every text architecture, followed by per-family capability notes for image, audio and PDF input, thinking mode, and tool calling.

Sentence encoders: Snowflake Arctic Embed L v2.0 / all-MiniLM-L6-v2, served through --embeddings and OpenAI/Ollama embedding APIs. Embedding models, supported backends, and validation →

Download & run, per family

Install and verify the .NET 10 SDK for your platform before running a build command. The first block is the verified Gemma 4 E4B quick start. All blocks use the Hugging Face CLI (pip install -U huggingface_hub); the per-family blocks after the first assume a full source build from the repository root. One-shot CLI text prompts are read from a file via --input (put your question in prompt.txt first — --prompt is reserved for the image- and video-generation models). Swap --backend for ggml_metal / ggml_vulkan / ggml_cpu to match your hardware (the CLI default is ggml_cpu).

Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)

Copying and running the commands takes about 30 seconds; the 7.48 GiB download and the first restore/build take longer and depend on your connection and machine. Repository benchmarks verify TensorSharp's E4B Q8_0 family and execution path; the linked ggml-org repository is the recommended public source. This block is for Linux + NVIDIA:

hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
echo "Explain why local inference is useful." > prompt.txt

# Text needs no mmproj.
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet run --project TensorSharp.Server.Host -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan. For image, video, or audio input, also run hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models and pass --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The server's browser UI is http://localhost:5000/; its liveness endpoint is /health. See Getting Started for Windows PowerShell and full platform syntax.

DeepSeek V4.1 Flash (384 routed experts, text + image/video, thinking, tools)

V4.1 is not V4 with new weights — it differs on every forward pass, and renaming the GGUF architecture to deepseek4 is invalid. TensorSharp gives it its own native graph: four residual streams with delayed hyper-connection mixing, a 128-token raw sliding window per attention block (the first two layers have no compressed attention, the next 18 compress at ratio 2 and the last 20 at ratio 1), a lightning indexer with candidate block filtering, and Engram n-gram features injected at layers 1 and 14. ggml_cuda is the serving backend. ggml_cpu runs the same graph on scalar fallbacks so the architecture can be checked without a GPU; ggml_vulkan and ggml_metal need TS_DSV41_ALLOW_NON_CUDA_GPU=1 and pay a host round trip per architecture-specific op; cpu runs a pure-C# V4.1 executor — no ggml, no native library and no GPU — held to the PyTorch reference at 2e-5 on a small synthetic fixture (architectural agreement, not parity on the real Q2_K weights), and text only there because the vision companion is a native ggml component; cuda runs V4.1 on the direct-CUDA engine's own kernels; both are correctness and portability paths, not serving ones; mlx refuses the checkpoint rather than loading V4.1 weights into a graph that does not implement it.

New downloads: use the repaired smalinin/DeepSeek-V4.1-Flash-GGUF Q2_K-Q5/ set below: ten shards totaling 312.349 GiB, with F32 mHC matrices, BF16 Engram gates and Q5_K Engram tables. The original seven-shard Q2_K checkpoint quantized sensitive matrices and is no longer recommended for quality validation. Historical seven-shard throughput is kept separate from qualification of this repaired checkpoint. Model card and file checksums →

The current published GGUF already includes Engram weights and hash constants. TensorSharp reads them directly; text inference needs no Engram preparation or separate Engram file. Images and video still need the optional vision companion because the conversion omits the vision tower, aligner and visual routing biases.

# Repaired Q2_K/Q5_K: ten shards, 312.349 GiB — keep the Q2_K-Q5 directory intact
python3 -m venv /tmp/dsv41-tools
/tmp/dsv41-tools/bin/python -m pip install huggingface_hub
/tmp/dsv41-tools/bin/hf download smalinin/DeepSeek-V4.1-Flash-GGUF \
    --revision d1de55c19f95172c882906cc83c0e55932d26a63 \
    --include "Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-*.gguf" --local-dir models/deepseek41-q2-q5

# Optional: the ~970 MB vision companion that --mmproj needs
/tmp/dsv41-tools/bin/python -m pip install numpy==2.0.2 gguf
/tmp/dsv41-tools/bin/python eng/dsv41-prepare-vision.py models/deepseek41-q2-q5/Q2_K-Q5 \
    --parent-model models/deepseek41-q2-q5/Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf \
    --repository deepseek-ai/DeepSeek-V4.1-Flash \
    --revision dba1be0a40aa45a94ad051997016db3960a90277

# Served over eight GPUs — --layer-split here is a LAYER SPLIT, not tensor parallelism
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/deepseek41-q2-q5/Q2_K-Q5/DeepSeek-V4.1-Flash-EngramQ5-Q2_K-00001-of-00010.gguf \
    --mmproj models/deepseek41-q2-q5/Q2_K-Q5/deepseek41.vision.gguf \
    --backend ggml_cuda --layer-split 8 --port 5000

For text-only serving, omit the vision preparation commands and --mmproj. The throughput figures below describe earlier binaries and checkpoints; see the validation report for the revision and scope of each measurement.

The historical vcruz305 eleven-shard Q4_K_M checkpoint (415 GiB) also include their Engram constants. There the two Engram tables grow to 51.5 GiB each; on eight 46 GB cards they stay host mappings and the routed experts need CPU offload — the loader prints the --n-cpu-moe N it wants and refuses to start rather than running out of VRAM mid-prompt.

For the historical seven-shard Q2_K run, the two Engram tables are GPU-resident by default when they fit, gathered in-graph with get_rows over the quantized rows, because TensorSharp keeps them in the checkpoint's own quantization — a Q2_K row of 256 values is 84 bytes, so both 384M-row tables together are 60.1 GiB. Historical measurements on eight A40s with a cold prompt each time: 532–541 prefill / 35.6–35.9 decode tok/s GPU-resident, against 506–528 / 34.5–35.2 for host mappings warmed with TS_DSV41_ENGRAM_WARM=1 and 207–221 / 28.0–29.2 unwarmed. The GPU-resident path also removes a 130-second whole-table warm at startup and the 60 GiB of host page cache the warmed path depends on. Placement is conservative: if putting the tables on GPUs would force routed-expert CPU offload, they stay host mappings and startup says so.

--layer-split N selects local whole-layer placement. --tp N selects experimental routed-MoE tensor parallelism over the same GPUs — gate/up split along the FFN intermediate, down along its input, partials reduced through host-staged F32 buffers — and the historical seven-shard Q2_K run was slower than layer split; that result does not qualify the repaired ten-shard checkpoint. Concurrent requests get isolated sequence slots; native ggml_cuda can decode their tokens in one graph, while other native backends use per-slot forwards. Batched GEMM shapes can change near-tie greedy choices; TS_BATCHED_FUSED_DECODE=0 selects serial decode. DSpark disables batched decode and retained-cache reuse. DSpark on V4.1 is experimental: the loader accepts a deepseek41-dspark drafter through --draft-model on ggml_cuda or ggml_cpu; initial text/image HTTP probes with trained weights passed using two-GPU layer split on ggml_cuda; broad quality and throughput remain unqualified — while V4 drafters are still rejected rather than half-loaded. Audio-bearing requests are refused with HTTP 400 instead of being silently dropped. What has been measured, and what explicitly has not, is in the validation report.

DeepSeek V4 Flash (284B MoE, text, thinking, tools, DSpark)

A 284B mixture-of-experts model with a 128-token raw sliding window plus block-compressed attention (1M advertised context). It does not use the generic per-op forward: TensorSharp runs it through one of three dedicated whole-model executors — --backend cuda (direct CUDA, no ggml), --backend ggml_cuda / ggml_vulkan (native ggml), and --backend cpu (100% pure C#, no native dependencies). The GPU executors use --layer-split N to place whole layers across N local GPUs, allowing models larger than one card to run. Without a placement configuration they use one device; the CPU executor streams weights from the mapped shards.

# ~160 GB of weights for the Q8 tier — pick a smaller quant directory if that is too much
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models

# Plain decode — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --layer-split 4 --input prompt.txt --max-tokens 200

# DSpark block speculative decoding (~1.3-1.4x decode, measured with greedy sampling)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --layer-split 4 --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
    --input prompt.txt --max-tokens 200 --temperature 0

# Served over HTTP, 4 GPUs, with speculation on
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --layer-split 4 --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf

DSpark drafts a whole block of tokens per step and the trunk verifies the block in one batched forward, so greedy output is unchanged. On the CLI and the server alike every verify row is drawn with the run's or request's own sampler, so it composes with any sampling settings; under repetition, presence or frequency penalties acceptance falls, because the drafted block is not penalized. See DSpark speculative decoding.

GLM 5.x (744B-A40B MoE, text, thinking, tools)

A 744B mixture-of-experts model — 256 routed experts at top-8 plus one shared expert — built on DeepSeek Sparse Attention: Multi-head Latent Attention with weight absorption, so the cache is one 576-wide row per token per layer, plus a "lightning indexer" that decides which cached tokens each query may attend to (1M advertised context). Like DeepSeek V4 it skips the generic per-op forward: --backend ggml_cuda / ggml_vulkan / ggml_cpu / ggml_metal run a native whole-model ggml executor that accepts --layer-split N on CUDA or Vulkan to place 226 GiB of weights across N local GPUs (default: one device), while --backend cpu (100% pure C#, no native dependencies) and --backend cuda run the managed per-op path. MLX does not run it. The multi-shard GGUF is read by GgufFile itself, so --model points at the first shard. The commands below are GLM-5.2's; GLM-5.3 (not Flash) shares the same glm-dsa loader and gets its own block after them.

# ~226 GiB for the UD-IQ2_XXS tier (6 shards) — pick a different quant directory for more or less
hf download unsloth/GLM-5.2-GGUF --include "UD-IQ2_XXS/*" --local-dir models

# Explicit local layer split on 3x96 GB GPUs — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --layer-split 3 --input prompt.txt --max-tokens 200

# Thinking mode, interactive
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --layer-split 3 --interactive --think

# Not enough VRAM? Keep the routed experts of the first N layers in system RAM
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --layer-split 3 --n-cpu-moe 30 --input prompt.txt --max-tokens 200

# Served over HTTP
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --layer-split 3

The advertised 1M-token context is a ceiling, not a promise: 1M tokens of MLA cache is about 93 GiB, so once the weights land the loader asks the devices how much VRAM is actually free, sizes the context to what fits alongside one full prefill graph, and logs what it picked. On 3× RTX PRO 6000 with GLM-5.2 that is 342,272 tokens on the plain layer split, 646,400 with --n-cpu-moe 30, and 91,136 under --tp 3 (every rank holds a full-length cache). Set MAX_CONTEXT to make a specific length a hard requirement instead — it is honoured if it fits and refused with the numbers if it does not.

--tp N works here too, but on PCIe-attached cards it is a capacity feature rather than a speed one: the two all-reduces every one of the 78 layers needs cost more bus time than the split saves, so with GLM-5.2 --tp 3 measures pp2048 505.6 / tg64 17.6 tok/s against 915.9 / 43.9 on the layer split. Concurrency is served by native per-sequence slots (each request owns its MLA and indexer caches), with a default-on batched fused decode worth 1.81× aggregate at 4 concurrent requests (measured on GLM-5.2); set TS_BATCHED_FUSED_DECODE=0 to disable it. See Tensor parallelism and Continuous batching.

GLM-5.3 (not Flash) is the same glm-dsa block shape as GLM-5.2 — 79 blocks (78 trunk + one NextN), 256 routed experts at top-8 plus one shared expert, MLA with the lightning indexer, rope base 8e6 — so it loads on the GLM-5.2 path with no new code and no new flag: one architecture descriptor serves GLM-5.2, GLM-5.3 and GLM-5.3-Flash, and Flash versus non-Flash is a single boolean on general.architecture. unsloth/GLM-5.3-GGUF keeps one subdirectory per quant, each a multi-shard set — UD-Q2_K_XL is seven shards, 236.4 GiB. It is text only twice over: the repository publishes no mmproj at any quant, and an --mmproj handed to glm-dsa is warned about and ignored rather than refused, so the result is a text-only run and not a failure. --spec loads the complete NextN block at blk.78, but that block ships no nextn.shared_head_head.weight and borrows the trunk LM head, which is column-parallel under --tp — so speculation is engaged on single-device or explicit --layer-split N placement (no active tensor parallelism), and the flag has to be on the command line before the model loads.

# ~236.4 GiB for the UD-Q2_K_XL tier (7 shards) — pick a different quant directory for more or less
hf download unsloth/GLM-5.3-GGUF --include "UD-Q2_K_XL/*" --local-dir models

# Explicit local layer split on 8x46 GB GPUs — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-UD-Q2_K_XL-00001-of-00007.gguf \
    --backend ggml_cuda --layer-split 8 --input prompt.txt --max-tokens 200

# NextN speculation — use explicit layer split here, with no active tensor parallelism
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-UD-Q2_K_XL-00001-of-00007.gguf \
    --backend ggml_cuda --layer-split 8 --spec

Measured on 8× A40 46 GB without NVLink, GLM-5.3 UD-Q2_K_XL, a 10,531-token prompt, 300 decode tokens, median of three, whole-layer placement: prefill 251.6 t/s, decode 20.48 t/s and a 264 s load, against llama.cpp's 20.28 t/s decode and 753 s load on the same box — a decode tie with a 2.9× faster load of a 236.4 GiB checkpoint. The honest gap is time to first token: 41.9 s against 29.0 s, about 1.4× slower (that llama.cpp cell recorded no prompt-token count, so no prefill tok/s is reported for it). --tp N is accepted here but local / single-process only, and it has never been run on GLM-5.3: every rank would hold a full replicated KV cache, and the only recorded arithmetic is a non-fit — --tp 8 wants 41.7 GiB per rank against 46 GB cards. --tp-node-id / --tp-peers are refused for the whole GLM family before the model is built. Both releases share the GLM model card.

GLM-5.3-Flash (320B MoE, text + image, thinking, tools)

The hybrid successor loads through the same native whole-model ggml executor and the same model class as GLM-5.2 — its GGUF architecture id is glm5next — with four architectural changes layered on top. 320B parameters, 288 routed experts at top-8 plus one shared, 46 blocks = 45 trunk + 1 NextN. KDA linear attention runs on 34 of the 45 trunk layers; the other 11 are NoPE MLA + DSA (no rope anywhere in the text tower, the 512-wide latent is the cache row); the indexer is pooled — 4-cell pools, top-k 2048 taken over pools and then expanded to their members; hyper-connections are Sinkhorn over ×4 streams; and every SwiGLU is clamped at limit 10. The KDA recurrent state cannot be rewound, so a cached prefix is reused only when the new prompt extends it exactly.

# ~101 GiB for the UD-Q2_K_XL tier — pick a different quant directory for more or less
hf download unsloth/GLM-5.3-Flash-GGUF --include "UD-Q2_K_XL/*" --local-dir models
hf download unsloth/GLM-5.3-Flash-GGUF mmproj-BF16.gguf --local-dir models

# Explicit local layer split on 2x96 GB GPUs — point --model at the FIRST shard
# (N is however many shards the quant directory you downloaded holds)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
    --backend ggml_cuda --layer-split 2 --input prompt.txt --max-tokens 200

# Image understanding — the GLM-OCR ViT ships as mmproj-BF16.gguf
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
    --max-tokens 300 --backend ggml_cuda --layer-split 2

# Not enough VRAM? Keep the routed experts of the first N layers in system RAM
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
    --backend ggml_cuda --layer-split 2 --n-cpu-moe 10 --input prompt.txt --max-tokens 200

# Served over HTTP
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/UD-Q2_K_XL/GLM-5.3-Flash-UD-Q2_K_XL-00001-of-0000N.gguf \
    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda --layer-split 2

What runs today: the whole-layer placement across N GPUs with --layer-split N, --cpu-moe / --n-cpu-moe N host-resident experts, per-sequence native slots for serving, and vision — --image, multi-image prompts and multi-turn image sessions, through the GLM-OCR ViT. On GGML GPU backends, --tp N selects native local/single-process tensor parallelism, as it does for GLM-5.2: KDA/MLA heads and routed-expert hidden rows are sharded, and KDA recurrent state is per-rank. With full sharding on one rank per local GPU, no CPU MoE or tracing, and native hyper-connection kernels, segmented rank-local graphs submit concurrently: routed-MoE partials reduce first, then every rank computes and adds its replicated shared expert locally before Sinkhorn hyper-connections; hyper-connections, pooled indexer, router, norms, dense layers and embedding remain unsharded and execute per rank, while output norm / LM head stay on rank 0. CPU MoE, tracing, partial sharding, oversubscription, or missing native hyper-connection kernels select the combined scheduler fallback, where the shared expert runs once on rank 0. The native TP path for both GLM-5.2 and GLM-5.3-Flash is local/single-process, not distributed or cross-node. Not yet: NextN/MTP speculation.

Measured on 2× RTX PRO 6000 Blackwell (96 GB) with GLM-5.3-Flash-UD-Q2_K_XL (101 GiB), layer split, both engines at n_ubatch 2048, back to back: pp2048 2070 t/s for llama.cpp against 2014 for TensorSharp, pp16384 1690 against 1692, pp32768 1483 against 1446, and tg64 36.6 against 73.5. Prefill is within a few percent either way; decode runs at 2.0× llama.cpp.

It runs on --backend cpu too — the 100% pure-C# path, no native dependencies — now that glm5next's NoPE MLA (no rope half anywhere, so nothing to split off) and its IQ2_XS / IQ4_XS weights have managed support. It is well behind the GGML backends there: on a 122-CPU box (22-token prompt, 16 tokens out) it measures prefill 3.1 / decode 1.6 tok/s against ggml_cpu's 17.7 / 3.9 on the same file and box — about 5.7× off on prefill and 2.4× on decode, with roughly 89% of the time in the MoE expert path. Treat it as a reference implementation to A/B against rather than a bit-parity one: the prefill-logit cosine against ggml_cpu is 0.9567 over the 154880-wide vocabulary and the greedy text differs, because native's top two logits are 0.11 apart and the managed path ranks them the other way. That is consistent with 2-bit expert-pick sensitivity, but it is not proven to be only that — 0.96 is lower than the ~0.999 a higher-precision checkpoint would be expected to give, and no higher-precision GLM-5.3-Flash GGUF was available as a control. The measurements, the four fixes it took, and the caveat in full are in the GLM model card.

Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)

hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda

For Qwen 3.6 NextN speculative decoding, download the trunk from the -MTP- repo (the base repo's GGUFs strip the NextN block and silently fall back to standard decode) and pass --spec to the CLI or the server — the explicit opt-in, because the NextN drafter is embedded in the trunk checkpoint and loading it pages extra weights into VRAM:

hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --spec

Qwen3.8-27B, a dense model, loads through the same qwen35 plug-in — for example Qwen3.8-27B-UD-Q4_K_XL.gguf from unsloth/Qwen3.8-27B-GGUF, which config/agent-qwen3.8-27b.json pins by commit and SHA-256 and which keeps the embedded NextN block for --spec. It also takes a DFlash2 block drafter, a separate GGUF (z-lab/Qwen3.8-27B-DFlash2-GGUF) attached with --draft-model, which replaces the NextN block when attached. What it gains depends on the workload: on an RTX 3080 Laptop with UD-IQ3_XXS, 1.14× on free-form prose and 1.63× on a highly predictable prompt — see §12.4 of the Qwen 3.5 card.

Qwen 3.8 Flash Next (hybrid MoE, text + image/video, thinking, tools, MTP head)

A hybrid mixture-of-experts model whose GGUF architecture id is qwen4exp: GatedDeltaNet recurrent layers interleaved with full-attention layers (some behind Qwen Sparse Attention's indexer), a PLE n-gram embedding block, ×4 hyper-connection streams and a 512-expert MoE. 48 layers, hidden size 2560, 24 query / 2 KV heads at head_dim 256, a 248,320-token vocabulary, 10 of the 512 experts used per token, GDN on 36 of the 48 layers and PLE on layer 1. On the GGML backends the whole token runs as (almost) one graph — embedding, in-graph PLE, all 48 layers, the final mixer and the LM head — replayed from a shape-keyed cache of captured graphs. Vision rides the Qwen3.5-VL tower with (T,H,W) IMRoPE positions, so multi-image prompts and multi-turn image sessions both work. A cached prefix is reused only when the new prompt extends it exactly (the GDN recurrence cannot rewind), and in the radix prefix cache that reuse stops at the conversation's first image or video span: a later turn restores at most a checkpoint that ends before the attachment, typically the system prompt, and re-prefills the rest. Concurrent requests are served through per-sequence state holders — each request owns its attention KV and indexer caches, its GDN and PLE state — so switching between them is a reference swap. Thinking is switchable, as on Qwen 3.5: --think / "think": true opens the <think> block, and without it the published template renders a closed, empty one.

# ~73.4 GiB for the UD-Q2_K_XL tier — pick a different quant directory for more or less
hf download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q2_K_XL/*" --local-dir models
hf download unsloth/Qwen3.8-Flash-Next-GGUF mmproj-BF16.gguf --local-dir models

# Text — point --model at the FIRST shard
# (N is however many shards the quant directory you downloaded holds)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 300

# Image understanding
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
    --mmproj models/mmproj-BF16.gguf --image photo.png --input question.txt \
    --max-tokens 300 --backend ggml_cuda

# Two GPUs: --layer-split N runs a LAYER SPLIT here, not tensor parallelism
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model models/UD-Q2_K_XL/Qwen3.8-Flash-Next-UD-Q2_K_XL-00001-of-0000N.gguf \
    --mmproj models/mmproj-BF16.gguf --backend ggml_cuda --layer-split 2

Local CUDA tensor parallelism is now available. --tp N on ggml_cuda splits routed/shared FFN intermediate channels and output rows, using FP32 gathers to preserve down-projection reduction order; attention, GDN, QSA and PLE weights/state stay replicated. The loader checks the degree, quantization and exact-strip kernel layout before bulk loading. Physical-device qualification covers UD-IQ4_XS on NVIDIA A40 TP2/TP4; UD-Q2_K_XL's IQ2_XS/IQ3_XXS FFNs and floating-point FFNs are refused under TP. Prefix checkpoints remain disabled in this mode. --layer-split N remains a separate GGML CUDA/Vulkan option, including for UD-Q2_K_XL; the direct cuda engine uses layer split. Do not combine the modes; multi-node groups are unsupported. The older 2× A100 UD-Q2_K_XL measurements concern layer split: identical greedy output, 24.2 + 26.2 GB VRAM, and about 1520–1550 prefill / 56 decode tok/s. They do not measure the new TP path. Quantization constraints, parity and timing scope →

Speculative decoding uses a separate shared MTP head, not tensors in the trunk (the published Q8_0 shards carry no nextn/mtp tensors): --draft-model mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf (2,786,568,256 bytes; unsloth/Qwen3.8-Flash-Next-GGUF at revision 38bb39ee) attaches it on a GGML backend. It drafts only for a solo request that prefilled from position 0; turns that continue a retained holder or a shared-prefix clone decode plainly. Measured on UD-Q2_K_XL over a three-GPU A40 layer split with --spec-draft 3, a 192-token code-copy stream was identical to plain greedy at 1.69× plain decode (83.2 against 49.1 tok/s) with the current verify-row kernels, which make every verify row bit-identical to its decode step (1.75–1.96× before that change on 2026-09-17), but prose does not pay: 512-token answers decoded at 44–46 tok/s against 52 plain. See the model card.

Larger than memory. UD-Q2_K_XL is 78.9 GB (73.4 GiB), but a token reads only 16 rows of its 28.8 GB n-gram table and 10 of each layer's 512 experts, so TensorSharp runs it on machines that cannot hold it. On ggml_metal the n-gram table stays in the GGUF memory mapping and a token's rows are read on demand; unless --n-cpu-moe N or --cpu-moe is given, the engine also plans the expert offload, keeping whole layers' experts on the GPU while they fit and running the first layers' on the host from the same mapping, where the page cache holds the hot ones and the rest come from the SSD as tokens need them. On an M5 Pro with 48 GB (33 of the 48 layers' experts on the host), a 1,818-token prompt with 256 greedy tokens measured 109.4–115.5 prefill / 12.8–12.9 decode tok/s against 20.4–22.4 / 13.00–13.24 for llama.cpp with -ngl 0 -t 12 (28.8–31.6 / 11.94–12.48 with --no-op-offload), whose full GPU offload runs out of memory there. --backend mlx is refused up front: MLX has no kernels for the sparse-attention indexer, the hyper-connections, the n-gram table or the IQ2_XS/IQ3_XXS experts. The direct cuda engine runs those experts: on 2× A40 with --layer-split 2, warm, the same prompt measured 1,612–1,613 / 56.85–56.93 tok/s (ggml_cuda: 1,174–1,217 / 52.9–53.1) against llama.cpp's 752–963 / 59.04–59.86. TensorAgent offers the file on 48 GB Macs; see the model card.

Bonsai2 27B (PRISM ternary Qwen 3.5, text + image)

Ternary-Bonsai-2-27B-PQ2_0.gguf and -PTQ1_0.gguf declare general.architecture = qwen35 — 64 layers, 48 GatedDeltaNet and 16 full attention — and add PRISM signed-Hadamard metadata (prism.hadamard.*) and the publisher tensor types PQ2_0 (142) and PTQ1_0 (143). TensorSharp recognizes them from that metadata, transcodes the weights losslessly to GGML Q2_0 at load (about 6% larger for PQ2_0 and 29% for PTQ1_0, so loaded memory is not the file size), and applies the declared rotations to projection inputs and embedding outputs; reading the weights without them produces the wrong network. It needs a single-device GGML backend: cpu, cuda, mlx and --tp are refused rather than silently skipping the rotations. The companion projectors Ternary-Bonsai-2-27B-mmproj-BF16.gguf / -mmproj-Q8_0.gguf add image input — the CLI looks for them beside the model, the server loads one only through --mmproj. The Bonsai2 card lists the files' exact bytes and SHA-256. They are published at prism-ml/Ternary-Bonsai-2-27B-gguf (Apache-2.0; pinned revision b072e1d), and TensorAgent offers the PTQ1_0 file, with the optional Q8_0 projector, as an experimental download on 16 GB devices.

# Cap the context rather than allocating the advertised 262k window
MAX_CONTEXT=4096 KV_CACHE_DTYPE=f16 \
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model /path/to/Ternary-Bonsai-2-27B-PQ2_0.gguf \
    --backend ggml_metal --input prompt.txt --max-tokens 128 --temperature 0

# Served, with a projector for image input
MAX_CONTEXT=4096 KV_CACHE_DTYPE=f16 \
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model /path/to/Ternary-Bonsai-2-27B-PTQ1_0.gguf \
    --mmproj /path/to/Ternary-Bonsai-2-27B-mmproj-Q8_0.gguf --backend ggml_metal

Validated on Metal (M5 Pro) only, on 2026-09-22: both formats reproduced the publisher llama.cpp fork's greedy output exactly on four 32-token prompts and passed HTTP, JSON, tool-call and image smokes (functional checks, not a quality evaluation). Against that fork, PQ2_0 measured 364.05 / 25.50 tok/s prefill / decode against 384.44 / 26.96, and PTQ1_0 364.49 / 25.53 against 357.67 / 26.31 — the card records its performance target as unmet. A CUDA-only defect in the signed-Hadamard path was fixed afterwards, but no CUDA results are committed: CUDA, Vulkan, iOS and tensor-parallel execution remain unvalidated.

GPT OSS (text, thinking always on, tools)

hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda

Nemotron-H (text; image on the Omni distribution)

hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
    --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda

For image input use the Nemotron 3 Nano Omni GGUF with its mmproj-BF16.gguf (table above). Audio needs the Parakeet/FastConformer audio tower, which no public GGUF ships: TensorSharp runs it from a separately prepared companion GGUF (conversion scripts in eng/nemotron-audio/), passed as --mmproj or through TS_NEMOTRON_AUDIO_MMPROJ beside a vision projector. Without one, audio is refused (HTTP 400 on the server, an error on the CLI) instead of being silently dropped. On CPU the tower matches the reference modules with F32 compute (the companion prepare_f32.py writes); on the trained weights the BF16-compute companion that prepare.py writes still fails that comparison. Spoken-audio answer quality and GPU execution are not qualified.

Nemotron 3.5 Lightning 30B-A3B loads through the same nemotron_h_moe path — 23 Mamba-2 + 23 MoE + 6 attention layers, e.g. NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MXFP4_MOE.gguf (~17 GB) from unsloth/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF. It does not speculate, and neither does any Nemotron-H: the trunk's multi-token verify and its single-token decode run different attention and MoE kernels, so a speculative stream would not match plain greedy decoding (the 2026-09-16 campaign with NVIDIA's DSpark drafter for this model saw every solo request diverge). A --draft-model is refused — the server stops at startup — and --spec serves plain decoding with a one-time warning. → how block drafting works on the families that take it

Hunyuan Dense (text only, single device)

Tencent's dense Hunyuan decoders — the checkpoints declaring general.architecture = hunyuan-dense, such as the Hy-MT2 releases. Before the architecture was registered these official files failed at load: the loader fails closed on an unknown architecture rather than guessing a nearby graph. The text graph matches llama.cpp's hunyuan-vl text tower — RMSNorm, fused QKV, NeoX RoPE, then per-head Q/K RMSNorm, GQA and SwiGLU. QK-norm runs after RoPE, the opposite of Qwen 3.5; getting that order wrong yields fluent nonsense rather than a load error, so the norms are mandatory and a layer missing them throws.

# Any GGUF whose general.architecture is hunyuan-dense — the file name is not what selects it
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/<your-hunyuan-dense>.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 200

First cut: text only, single device, generic per-op path, no thinking channel and no tool parser. Extra GPUs stay idle and startup says so rather than leaving it to be inferred.

Mistral 3 (text + image)

hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda

Muse-Glimmer (text + image, thinking, tools, DFlash drafting)

hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --input prompt.txt --max-tokens 256 --backend ggml_cuda
# image understanding
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \
    --max-tokens 300 --backend ggml_cuda
# DFlash speculative decoding (optional drafter GGUF from the same repo)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --draft-model models/dflash-kquant.gguf --spec-draft 15 --input prompt.txt --backend ggml_cuda

DiffusionGemma (block text diffusion, text + image)

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models
# Optional, for image input: the vision tower is in no GGUF, only in this upstream shard (2.84 GB)
hf download google/diffusiongemma-26B-A4B-it model-00011-of-00011.safetensors --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
# Ask about a picture (repeat --image for several, in order)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \
    --mmproj models/model-00011-of-00011.safetensors --image photo.png --input question.txt \
    --max-tokens 256 --backend ggml_cuda
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf \
    --mmproj models/model-00011-of-00011.safetensors --backend ggml_cuda

Every published GGUF of this checkpoint is text-only, so the Gemma 4 vision tower (gemma4v) is loaded straight from the upstream model-00011-of-00011.safetensors shard, or from an mmproj GGUF; neither host looks for it beside the model, and without it an image is refused rather than answered blind. config/diffusiongemma-26b-a4b-q4.json and -q3.json fetch the shard for you. Audio is refused — the checkpoint has no audio tower — and so is video; send frames as separate images. Prompts always render with thinking off: a thought block the model writes anyway is returned as reasoning only with "think": true. Requests that carry tools are refused with HTTP 400.

The same server also answers typed decisions on POST /v1/systemone, a Jev-compatible endpoint: noul (boolean), choice and score questions about a text or image state (up to 8 images), read from label probabilities after one denoising step instead of from generated JSON. jev-latest and jev-preview are aliases for the loaded model, not separate checkpoints. config/jev-diffusiongemma-q4.json downloads the model and the vision shard and binds 127.0.0.1:5000 on ggml_cuda; the request format is in the Jev card.

Multimodal support

Send images/audio/video via the CLI (--image, --video, --audio), the Web UI uploads, or the HTTP API (base64 images array for Ollama, image_url data URI for OpenAI). PDF documents are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's --pdf flag (one-shot mode) or the Web UI upload; on both, TS_PDF_MAX_PAGES caps the page count (default: all pages).

Thinking / reasoning mode

Thinking-capable models (Qwen 3.5/3.6, Qwen 3.8 Flash Next, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, DeepSeek V4.1, GLM 5.x, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.

• Qwen 3.5/3.6 / Nemotron-H — <think>…</think> tags. With thinking off, Nemotron-H Reasoning-128K sometimes reasons past its closed <think></think> and closes the block itself. When the close starts within the reply's first 2,048 characters, that text is reported as reasoning, not as the answer; a later close counts as answer text, so the reasoning and the literal </think> stay in the answer, a known limitation (details).
• Qwen 3.8 Flash Next — the same <think>…</think> tags and the same switch: with thinking off, the published template renders a closed, empty <think></think> block.
• Gemma 4 — <|channel>thought …<channel|> tags.
• GPT OSS — Harmony format: <|channel|>analysis for thinking, <|channel|>final for the answer.
• DeepSeek V4 — <think>…</think> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.
• DeepSeek V4.1 — the same tags, with a trained closing transition: at TS_THINKING_BUDGET the model emits </think> and finishes the answer inside the original max_tokens. The budget defaults to 75% of the output allowance when that allowance is at least 512 tokens.
• GLM 5.x — <think>…</think> tags, also opt-in: --think adds a Reasoning Effort: Max system line and leaves the block open for the model to close, and without it the prompt emits an empty <think></think>. Earlier turns' reasoning is always dropped from the prompt. GLM-5.3-Flash always reasons: its system line maps reasoning_effort: "low" / "high" to Low / High, with Max for omitted or "medium" effort, the generation prompt always opens <think>, and past turns keep their reasoning.
• Muse-Glimmer — an assistant to=self reasoning channel emitted by the chat template.
• DiffusionGemma — not prompted: prompts always render with thinking off. A thought block the model writes anyway is parsed out and returned as reasoning only with "think": true.

Enable it via --think (CLI), "think": true (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. message.thinking in the Ollama chat response.

Tool calling / function calling

Supported model families can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (CLI) or the tools parameter (API). Each supported architecture uses its own wire format, and its output parser extracts calls into structured tool_calls:

• Nemotron-H — Reasoning-128K: the Nemotron family convention, <TOOLCALL>[{"name": …, "arguments": {…}}]</TOOLCALL> (Python-style calls such as [read_file(path="notes.txt")] are accepted too). Nemotron 3 Nano / Omni: <tool_call> with an XML <function=NAME><parameter=key> body (the JSON form is still accepted).
• Qwen 3.5 / 3.6, Qwen 3.8 Flash Next — the same <tool_call> block with an XML body: <function=NAME><parameter=key>value</parameter></function> (the JSON form is still accepted).
• Gemma 4 — <|tool_call>call:function_name{args}<tool_call|>
• GPT OSS (Harmony) — tools declared as a TypeScript namespace; calls emitted on the commentary channel.
• Muse-Glimmer — ATEM XML markup declared by the chat template; the parser lifts the calls into the same structured tool_calls shape.
• DeepSeek V4.1 — spaced DSML (<｜DSML｜ calls>), which V4's unspaced form is not compatible with. On /v1/chat/completions a request-local grammar enforces the declared tools at the token level: tool_choice auto / required / none / a named function and parallel_tool_calls: false are all honoured, and unsupported schema assertions are refused with HTTP 400 before generation.
• DeepSeek V4 — DSML markup: the system prompt teaches the syntax and carries one JSON schema per function, and the model answers with <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>….
• GLM 5.x — XML with per-argument tags: <tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value>…</tool_call>. The function name follows the opening tag as bare text and every argument is its own key/value pair; values the template rendered with tojson are parsed back into numbers, arrays and objects.

Mistral 3, Hunyuan Dense and DiffusionGemma are not on this list: the first two render neither tool declarations nor tool results, and DiffusionGemma refuses requests that carry tools, so client tools, built-in skill/code tools and sub-agent delegation are not offered to them. Selected Agent Skill instructions are inlined as a no-tool fallback.

See Tool calling over HTTP for a complete request/response example and the continuation loop.