Text & LLM Models

Copy-paste download and run recipes for every text architecture, followed by the capabilities they share: image, audio and PDF input, thinking mode, and tool calling.

Download & run, per family

Install and verify the .NET 10 SDK for your platform before running a build command. The first block is the verified Gemma 4 E4B quick start. All blocks use the Hugging Face CLI (pip install -U huggingface_hub); the per-family blocks after the first assume a full source build from the repository root. One-shot CLI text prompts are read from a file via --input (put your question in prompt.txt first — --prompt is reserved for Qwen-Image-Edit). Swap --backend for ggml_metal / ggml_vulkan / ggml_cpu to match your hardware (the CLI default is ggml_cpu).

Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)

Copying and running the commands takes about 30 seconds; the 7.48 GiB download and the first restore/build take longer and depend on your connection and machine. Repository benchmarks verify TensorSharp's E4B Q8_0 family and execution path; the linked ggml-org repository is the recommended public source. This block is for Linux + NVIDIA:

hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
echo "Explain why local inference is useful." > prompt.txt

# Text needs no mmproj.
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan. For image, video, or audio input, also run hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models and pass --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The server's browser UI is http://localhost:5000/index.html. See Getting Started for Windows PowerShell and full platform syntax.

DeepSeek V4 Flash (284B MoE, text, thinking, tools, DSpark)

A 284B mixture-of-experts model with a 128-token raw sliding window plus block-compressed attention (1M advertised context). It does not use the generic per-op forward: TensorSharp runs it through one of three dedicated whole-model executors — --backend cuda (direct CUDA, no ggml), --backend ggml_cuda / ggml_vulkan (native ggml), and --backend cpu (100% pure C#, no native dependencies). All of them layer-split the weights across every visible GPU, so a model far bigger than one card still runs; --tp N (or TS_DSV4_NGPU) caps how many GPUs it uses.

# ~160 GB of weights for the Q8 tier — pick a smaller quant directory if that is too much
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models

# Plain decode — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 200

# DSpark block speculative decoding (~1.3-1.4x decode; needs greedy sampling)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
    --input prompt.txt --max-tokens 200 --temperature 0

# Served over HTTP, 4 GPUs, with speculation on
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf

DSpark drafts a whole block of tokens per step and the trunk verifies the block in one batched forward, so greedy output is unchanged. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p, or penalty turns it off); on the server every verify row is drawn with the request's own sampler, so it composes with any sampling settings. See DSpark speculative decoding.

GLM 5.x (744B-A40B MoE, text, thinking, tools)

A 744B mixture-of-experts model — 256 routed experts at top-8 plus one shared expert — built on DeepSeek Sparse Attention: Multi-head Latent Attention with weight absorption, so the cache is one 576-wide row per token per layer, plus a "lightning indexer" that decides which cached tokens each query may attend to (1M advertised context). Like DeepSeek V4 it skips the generic per-op forward: --backend ggml_cuda / ggml_vulkan / ggml_cpu / ggml_metal run a native whole-model ggml executor that layer-splits 226 GiB of weights across every visible GPU, while --backend cpu (100% pure C#, no native dependencies) and --backend cuda run the managed per-op path. MLX does not run it. The multi-shard GGUF is read by GgufFile itself, so --model points at the first shard.

# ~226 GiB for the UD-IQ2_XXS tier (6 shards) — pick a different quant directory for more or less
hf download unsloth/GLM-5.2-GGUF --include "UD-IQ2_XXS/*" --local-dir models

# Layer split across every visible GPU — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 200

# Thinking mode, interactive
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --interactive --think

# Not enough VRAM? Keep the routed experts of the first N layers in system RAM
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda --n-cpu-moe 30 --input prompt.txt --max-tokens 200

# Served over HTTP
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
    --model models/UD-IQ2_XXS/GLM-5.2-UD-IQ2_XXS-00001-of-00006.gguf \
    --backend ggml_cuda

The advertised 1M-token context is a ceiling, not a promise: 1M tokens of MLA cache is about 93 GiB, so once the weights land the loader asks the devices how much VRAM is actually free, sizes the context to what fits alongside one full prefill graph, and logs what it picked. On 3× RTX PRO 6000 that is 342,272 tokens on the plain layer split, 646,400 with --n-cpu-moe 30, and 91,136 under --tp 3 (every rank holds a full-length cache). Set MAX_CONTEXT to make a specific length a hard requirement instead — it is honoured if it fits and refused with the numbers if it does not.

--tp N works here too, but on PCIe-attached cards it is a capacity feature rather than a speed one: the two all-reduces every one of the 78 layers needs cost more bus time than the split saves, so --tp 3 measures pp2048 505.6 / tg64 17.6 tok/s against 915.9 / 43.9 on the layer split. Concurrency is served by native per-sequence slots (each request owns its MLA and indexer caches), and TS_BATCHED_FUSED_DECODE=1 opts into a batched fused decode worth 1.81× aggregate at 4 concurrent requests. See Tensor parallelism and Continuous batching.

Gemma 3 (text + image)

hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda

Qwen 3 (text, thinking, tools)

hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda

Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)

hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda

For Qwen 3.6 NextN speculative decoding, download the trunk from the -MTP- repo (the base repo's GGUFs strip the NextN block and silently fall back to standard decode) and start the server with --mtp-spec — the MTP flags exist only on TensorSharp.Server:

hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec

GPT OSS (text, thinking always on, tools)

hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-MXFP4.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-MXFP4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-MXFP4.gguf --backend ggml_cuda

Nemotron-H (text; image on the Omni distribution)

hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
    --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda

For image input use the Nemotron 3 Nano Omni GGUF with its mmproj-BF16.gguf (table above). Audio inference is not functional — the required Parakeet audio mmproj is not shipped with the GGUF distribution.

Mistral 3 (text + image)

hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda

Muse-Glimmer (text + image, thinking, tools, DFlash drafting)

hf download unsloth/Muse-Glimmer-30B-GGUF Muse-Glimmer-30B-UD-IQ2_XXS.gguf --local-dir models
hf download unsloth/Muse-Glimmer-30B-GGUF mmproj-Muse-Glimmer-30B-Q8_0.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --input prompt.txt --max-tokens 256 --backend ggml_cuda
# image understanding
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --mmproj models/mmproj-Muse-Glimmer-30B-Q8_0.gguf --image photo.png --input question.txt \
    --max-tokens 300 --backend ggml_cuda
# DFlash speculative decoding (optional drafter GGUF from the same repo)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Muse-Glimmer-30B-UD-IQ2_XXS.gguf \
    --draft-model models/dflash-kquant.gguf --spec-draft-n-max 15 --input prompt.txt --backend ggml_cuda

DiffusionGemma (block text diffusion)

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda

Multimodal support

Send images/audio/video via the CLI (--image, --video, --audio), the Web UI uploads, or the HTTP API (base64 images array for Ollama, image_url data URI for OpenAI). PDF documents are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's --pdf flag (one-shot mode) or the Web UI upload (TS_PDF_MAX_PAGES caps the page count; default: all pages).

Thinking / reasoning mode

Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4, GLM 5.x, Muse-Glimmer) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.

• Qwen 3 / Qwen 3.5/3.6 / Nemotron-H — <think>…</think> tags.
• Gemma 4 — <|channel>thought …<channel|> tags.
• GPT OSS — Harmony format: <|channel|>analysis for thinking, <|channel|>final for the answer.
• DeepSeek V4 — <think>…</think> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.
• GLM 5.x — <think>…</think> tags, also opt-in: --think adds a Reasoning Effort: Max system line and leaves the block open for the model to close, and without it the prompt emits an empty <think></think>. Earlier turns' reasoning is always dropped from the prompt.
• Muse-Glimmer — an assistant to=self reasoning channel emitted by the chat template.

Enable it via --think (CLI), "think": true (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. message.thinking in the Ollama chat response.

Tool calling / function calling

Models can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (CLI) or the tools parameter (API). Each architecture uses its own wire format, but the output parser extracts calls into structured tool_calls regardless:

• Qwen 3 / Nemotron-H — <tool_call>{"name": …, "arguments": {…}}</tool_call>
• Qwen 3.5 / 3.6 — the same <tool_call> block with an XML body: <function=NAME><parameter=key>value</parameter></function> (the JSON form is still accepted).
• Gemma 4 — <|tool_call>call:function_name{args}<tool_call|>
• GPT OSS (Harmony) — tools declared as a TypeScript namespace; calls emitted on the commentary channel.
• Muse-Glimmer — ATEM XML markup declared by the chat template; the parser lifts the calls into the same structured tool_calls shape.
• DeepSeek V4 — DSML markup: the system prompt teaches the syntax and carries one JSON schema per function, and the model answers with <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>….
• GLM 5.x — XML with per-argument tags: <tool_call>NAME<arg_key>k</arg_key><arg_value>v</arg_value>…</tool_call>. The function name follows the opening tag as bare text and every argument is its own key/value pair; values the template rendered with tojson are parsed back into numbers, arrays and objects.

See Tool calling over HTTP for a complete request/response example and the continuation loop.