Supported Models

TensorSharp loads models in GGUF format and auto-detects the architecture from the file's general.architecture metadata. Pick a quantization that fits your hardware (Q4_K_M for low memory, Q8_0 for higher quality).

📘

Gemma 4 E4B is the example model in From Tensors to Tokens. Read the book for a guided build from tensor fundamentals through multimodal inference; use this page for current downloads and capability details. Explore the book →

Supported architectures

Architecture	GGUF arch keys	Example models	Multimodal	Thinking	Tools	MTP spec
DeepSeek V4 Flash	`deepseek4`	DeepSeek-V4-Flash (284B MoE, 256 experts, compressed sparse attention, 1M context)	Text only	Yes	Yes (DSML)	Yes (DSpark block drafter, separate GGUF)
Gemma 4	`gemma4`	gemma-4-E4B, 12B, 31B, 26B-A4B (MoE)	Image, Video, Audio	Yes	Yes	Yes (separate draft)
Gemma 3	`gemma3`	gemma-3-4b	Image	No	No	—
Qwen 3	`qwen3`	Qwen3-4B	Text only	Yes	Yes	—
Qwen 3.5 / 3.6	`qwen35`, `qwen35moe`, `qwen3next`	Qwen3.5-9B, Qwen3.5/3.6-35B-A3B (MoE)	Image	Yes	Yes	Yes on 3.6 (embedded NextN — only in GGUFs that retain the NextN block, e.g. the `-MTP-` repos)
GPT OSS	`gptoss`, `gpt-oss`	gpt-oss-20b (MoE)	Text only	Yes (always)	Yes	—
Nemotron-H	`nemotron_h`, `nemotron_h_moe`	Nemotron-H-8B, 47B, Nemotron 3 Nano Omni	Image (Omni)	Yes	Yes	—
Mistral 3	`mistral3`	Mistral-Small-3.1-24B-Instruct	Image	No	No	—
DiffusionGemma	`diffusion-gemma`, `diffusion_gemma`	diffusion-gemma text-diffusion GGUFs	Text only	No	No	—
Qwen-Image-Edit	`qwen_image`, `qwen-image`	qwen-image-edit MMDiT (+ VAE & Qwen2.5-VL companions)	Image edit (image+text → image)	No	No	—

Detailed per-model architecture cards (forward graph, components, parameters, and how TensorSharp optimizes prefill/decode) live under docs/models/ in the repository.

Model downloads (GGUF)

Architecture	Model	Download	Companion files (mmproj / MTP draft)
DeepSeek V4	DeepSeek-V4-Flash-0731 (284B MoE)	unsloth/DeepSeek-V4-Flash-0731-GGUF	One subdirectory per quant (`UD-Q8_K_XL/`, `UD-IQ4_XS/`, `UD-IQ1_S/`, …), each a multi-shard set — point `--model` at the `-00001-of-` shard. Text only; weights layer-split across every visible GPU
DeepSeek V4	DSpark speculative drafter (optional)	bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF	`DSpark-drafter-Q2K-Q8-0731.gguf` (7.0 GB) for the 0731 release, loaded with `--draft-model` for ~1.3–1.4× decode. Two other publishers' builds (5.6 GB / 10.9 GB) also load as-is — see `MODEL_DOWNLOADS.md`. Drafters for other architectures are a different design and are not supported
Gemma 4	gemma-4-E4B-it	ggml-org/gemma-4-E4B-it-GGUF	Recommended public artifact for the verified E4B Q8_0 native-GGML tier: `gemma-4-E4B-it-Q8_0.gguf`. Modalities: `mmproj-gemma-4-E4B-it-Q8_0.gguf` (same repo). MTP draft: AtomicChat/gemma-4-E4B-it-assistant-GGUF
Gemma 4	gemma-4-12B-it (QAT)	unsloth/gemma-4-12B-it-qat-GGUF	mmproj: `mmproj-BF16.gguf`; MTP draft: `mtp-gemma-4-12B-it.gguf` (both in the same repo)
Gemma 4	gemma-4-31B-it	ggml-org/gemma-4-31B-it-GGUF	mmproj: `mmproj-gemma-4-31B-it-Q8_0.gguf` (same repo)
Gemma 4	gemma-4-26B-A4B-it (MoE)	ggml-org/gemma-4-26B-A4B-it-GGUF	mmproj: `mmproj-gemma-4-26B-A4B-it-Q8_0.gguf` (same repo)
Gemma 4	gemma-4-26B-A4B-it (MoE, QAT)	unsloth/gemma-4-26B-A4B-it-qat-GGUF	mmproj: `mmproj-BF16.gguf`; MTP draft: `mtp-gemma-4-26B-A4B-it.gguf` (same repo). Alt draft: AtomicChat/gemma-4-26B-A4B-it-assistant-GGUF — pair drafts only with the matching target size
Gemma 3	gemma-3-4b-it (QAT, official)	google/gemma-3-4b-it-qat-q4_0-gguf	mmproj: `mmproj-model-f16-4B.gguf` (same repo). ⚠️ Gated: downloads require a Hugging Face login + accepting Google's Gemma license
Gemma 3	gemma-3-4b-it (non-gated alternative)	ggml-org/gemma-3-4b-it-GGUF	mmproj: `mmproj-model-f16.gguf` (same repo)
Qwen 3	Qwen3-4B	Qwen/Qwen3-4B-GGUF	— (text only)
Qwen 3.5 / 3.6	Qwen3.5-9B	unsloth/Qwen3.5-9B-GGUF	mmproj: `mmproj-F16.gguf` (same repo)
Qwen 3.5 / 3.6	Qwen3.5-35B-A3B (MoE)	ggml-org/Qwen3.5-35B-A3B-GGUF	mmproj: `mmproj-Qwen3.5-35B-A3B-Q8_0.gguf` (same repo)
Qwen 3.5 / 3.6	Qwen3.6-35B-A3B (MoE, NextN MTP)	unsloth/Qwen3.6-35B-A3B-MTP-GGUF	mmproj: `mmproj-F16.gguf` (same repo). These GGUFs retain the embedded NextN block for `--mtp-spec`; the base-repo GGUFs (unsloth/Qwen3.6-35B-A3B-GGUF) strip it and silently fall back to standard decode
GPT OSS	gpt-oss-20b (MoE)	ggml-org/gpt-oss-20b-GGUF	— (text only)
Nemotron-H	Nemotron-H-8B-Reasoning-128K	bartowski/nvidia_Nemotron-H-8B-…	— (text only)
Nemotron-H	Nemotron-H-47B-Reasoning-128K	bartowski/nvidia_Nemotron-H-47B-…	— (text only)
Nemotron-H	Nemotron 3 Nano Omni 30B-A3B (image)	unsloth/NVIDIA-Nemotron-3-Nano-Omni-…	mmproj: `mmproj-BF16.gguf` (same repo) — required for image input. Audio is not functional: it needs a Parakeet audio mmproj that this distribution does not ship
Mistral 3	Mistral-Small-3.1-24B-Instruct	bartowski/mistralai_Mistral-Small-3.1-24B-…	Pixtral mmproj: `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf` (same repo)
DiffusionGemma	diffusiongemma-26B-A4B-it	unsloth/diffusiongemma-26B-A4B-it-GGUF	—
Qwen-Image-Edit	MMDiT DiT (the `--model` GGUF)	unsloth/Qwen-Image-Edit-2511-GGUF	e.g. `qwen-image-edit-2511-Q4_K_M.gguf`
Qwen-Image-Edit	Qwen-Image VAE (required)	QuantStack/Qwen-Image-Edit-GGUF	`VAE/Qwen_Image-VAE.safetensors` — place next to the DiT or pass `--qwen-image-vae`
Qwen-Image-Edit	Qwen2.5-VL-7B text encoder (required)	unsloth/Qwen2.5-VL-7B-Instruct-GGUF	Optional vision mmproj: `mmproj-BF16.gguf` (same repo) for image-grounded edits
Qwen-Image-Edit	Lightning LoRA (optional, 4/8-step)	lightx2v/Qwen-Image-Edit-2511-Lightning	`Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors` via `--qwen-image-lora`

🧩

Multimodal models need a projector (mmproj) file. Download the matching file and pass its exact path with --mmproj. The server never auto-detects it, and CLI auto-detection recognizes only a small set of legacy filenames.

Download & run, per family

Install and verify the .NET 10 SDK for your platform before running a build command. The first block is the verified Gemma 4 E4B quick start. All blocks use the Hugging Face CLI (pip install -U huggingface_hub); the per-family blocks after the first assume a full source build from the repository root. One-shot CLI text prompts are read from a file via --input (put your question in prompt.txt first — --prompt is reserved for Qwen-Image-Edit). Swap --backend for ggml_metal / ggml_vulkan / ggml_cpu to match your hardware (the CLI default is ggml_cpu).

Quick start in ~30 seconds — Gemma 4 E4B Q8_0 (native GGML)

Copying and running the commands takes about 30 seconds; the 7.48 GiB download and the first restore/build take longer and depend on your connection and machine. Repository benchmarks verify TensorSharp's E4B Q8_0 family and execution path; the linked ggml-org repository is the recommended public source. This block is for Linux + NVIDIA:

hf download ggml-org/gemma-4-E4B-it-GGUF gemma-4-E4B-it-Q8_0.gguf --local-dir models
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
echo "Explain why local inference is useful." > prompt.txt

# Text needs no mmproj.
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf \
    --backend ggml_cuda

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan. For image, video, or audio input, also run hf download ggml-org/gemma-4-E4B-it-GGUF mmproj-gemma-4-E4B-it-Q8_0.gguf --local-dir models and pass --mmproj models/mmproj-gemma-4-E4B-it-Q8_0.gguf. The server's browser UI is http://localhost:5000/index.html. See Getting Started for Windows PowerShell and full platform syntax.

DeepSeek V4 Flash (284B MoE, text, thinking, tools, DSpark)

A 284B mixture-of-experts model with a 128-token raw sliding window plus block-compressed attention (1M advertised context). It does not use the generic per-op forward: TensorSharp runs it through one of three dedicated whole-model executors — --backend cuda (direct CUDA, no ggml), --backend ggml_cuda / ggml_vulkan (native ggml), and --backend cpu (100% pure C#, no native dependencies). All of them layer-split the weights across every visible GPU, so a model far bigger than one card still runs; --tp N (or TS_DSV4_NGPU) caps how many GPUs it uses.

# ~160 GB of weights for the Q8 tier — pick a smaller quant directory if that is too much
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF --include "UD-Q8_K_XL/*" --local-dir models
hf download bleysg/DeepSeek-V4-Flash-DSpark-drafter-GGUF DSpark-drafter-Q2K-Q8-0731.gguf --local-dir models

# Plain decode — point --model at the FIRST shard
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --input prompt.txt --max-tokens 200

# DSpark block speculative decoding (~1.3-1.4x decode; needs greedy sampling)
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf \
    --input prompt.txt --max-tokens 200 --temperature 0

# Served over HTTP, 4 GPUs, with speculation on
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
    --model models/UD-Q8_K_XL/DeepSeek-V4-Flash-0731-UD-Q8_K_XL-00001-of-00005.gguf \
    --backend ggml_cuda --tp 4 --mtp-spec --draft-model models/DSpark-drafter-Q2K-Q8-0731.gguf

DSpark drafts a whole block of tokens per step and the trunk verifies the block in one batched forward, so greedy output is unchanged. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p, or penalty turns it off); on the server every verify row is drawn with the request's own sampler, so it composes with any sampling settings. See DSpark speculative decoding.

Gemma 3 (text + image)

hf download ggml-org/gemma-3-4b-it-GGUF gemma-3-4b-it-Q4_K_M.gguf --local-dir models
hf download ggml-org/gemma-3-4b-it-GGUF mmproj-model-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --input prompt.txt --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gemma-3-4b-it-Q4_K_M.gguf --mmproj models/mmproj-model-f16.gguf --backend ggml_cuda

Qwen 3 (text, thinking, tools)

hf download Qwen/Qwen3-4B-GGUF Qwen3-4B-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3-4B-Q4_K_M.gguf --input prompt.txt --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3-4B-Q4_K_M.gguf --backend ggml_cuda

Qwen 3.5 / 3.6 (text + image, thinking, tools, NextN MTP on 3.6)

hf download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-UD-Q4_K_XL.gguf --local-dir models
hf download unsloth/Qwen3.5-9B-GGUF mmproj-F16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.5-9B-UD-Q4_K_XL.gguf --mmproj models/mmproj-F16.gguf --backend ggml_cuda

For Qwen 3.6 NextN speculative decoding, download the trunk from the -MTP- repo (the base repo's GGUFs strip the NextN block and silently fall back to standard decode) and start the server with --mtp-spec — the MTP flags exist only on TensorSharp.Server:

hf download unsloth/Qwen3.6-35B-A3B-MTP-GGUF Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --backend ggml_cuda --mtp-spec

GPT OSS (text, thinking always on, tools)

hf download ggml-org/gpt-oss-20b-GGUF gpt-oss-20b-mxfp4.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/gpt-oss-20b-mxfp4.gguf --input prompt.txt --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/gpt-oss-20b-mxfp4.gguf --backend ggml_cuda

Nemotron-H (text; image on the Omni distribution)

hf download bartowski/nvidia_Nemotron-H-8B-Reasoning-128K-GGUF nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --input prompt.txt \
    --think --max-tokens 400 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/nvidia_Nemotron-H-8B-Reasoning-128K-Q4_K_M.gguf --backend ggml_cuda

For image input use the Nemotron 3 Nano Omni GGUF with its mmproj-BF16.gguf (table above). Audio inference is not functional — the required Parakeet audio mmproj is not shipped with the GGUF distribution.

Mistral 3 (text + image)

hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf --local-dir models
hf download bartowski/mistralai_Mistral-Small-3.1-24B-Instruct-2503-GGUF mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf \
    --image photo.png --max-tokens 300 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/mistralai_Mistral-Small-3.1-24B-Instruct-2503-Q4_K_M.gguf \
    --mmproj models/mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf --backend ggml_cuda

DiffusionGemma (block text diffusion)

hf download unsloth/diffusiongemma-26B-A4B-it-GGUF diffusiongemma-26B-A4B-it-Q4_K_M.gguf --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --input prompt.txt \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0 --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/diffusiongemma-26B-A4B-it-Q4_K_M.gguf --backend ggml_cuda

Qwen-Image-Edit (image + prompt → edited image)

hf download unsloth/Qwen-Image-Edit-2511-GGUF qwen-image-edit-2511-Q4_K_M.gguf --local-dir models
hf download QuantStack/Qwen-Image-Edit-GGUF VAE/Qwen_Image-VAE.safetensors --local-dir models
hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf --local-dir models
hf download unsloth/Qwen2.5-VL-7B-Instruct-GGUF mmproj-BF16.gguf --local-dir models
hf download lightx2v/Qwen-Image-Edit-2511-Lightning Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors --local-dir models

dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf --image input.png \
    --prompt "Make the sky a dramatic sunset." --output edited.png \
    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --qwen-image-mmproj models/mmproj-BF16.gguf \
    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
    --backend ggml_cuda
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model models/qwen-image-edit-2511-Q4_K_M.gguf \
    --qwen-image-vae models/VAE/Qwen_Image-VAE.safetensors \
    --qwen-image-vl models/Qwen2.5-VL-7B-Instruct-Q4_K_M.gguf \
    --qwen-image-mmproj models/mmproj-BF16.gguf \
    --qwen-image-lora models/Qwen-Image-Edit-2511-Lightning-4steps-V1.0-bf16.safetensors \
    --backend ggml_cuda

Multimodal support

Family	Inputs	Notes
Gemma 4	Image · Video · Audio	Images PNG/JPEG/HEIC; Video MP4 (1 fps via OpenCV); Audio WAV 16 kHz mono / MP3 / OGG. E4B projector: `mmproj-gemma-4-E4B-it-Q8_0.gguf`.
Gemma 3	Image	PNG / JPEG / HEIC. Non-gated 4B projector: `mmproj-model-f16.gguf`.
Qwen 3.5 / 3.6	Image	Dynamic-resolution vision encoder. The 9B / 3.6 repositories use `mmproj-F16.gguf`.
Mistral 3	Image	Pixtral vision encoder. Projector: `mmproj-mistralai_Mistral-Small-3.1-24B-Instruct-2503-f16.gguf`.
Nemotron-H (Omni)	Image	RADIO / v2_vl ViT encoder. Pass the matching `--mmproj`; image tokens expand at `<image>` placeholders. Audio is preprocessed only — real audio inference needs a Parakeet audio mmproj that the GGUF distribution does not ship.

Send images/audio/video via the CLI (--image, --video, --audio), the Web UI uploads, or the HTTP API (base64 images array for Ollama, image_url data URI for OpenAI). PDF documents are supported too — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models — via the CLI's --pdf flag (one-shot mode) or the Web UI upload (TS_PDF_MAX_PAGES caps the page count; default: all pages).

Image editing (Qwen-Image-Edit)

Qwen-Image-Edit takes a prompt + an input image and returns an edited image — a different output modality from the text LLMs above. The loaded qwen_image GGUF is only the MMDiT (multimodal diffusion transformer); TensorSharp resolves two companion GGUFs alongside it:

Qwen-Image VAE — image ↔ 16-channel latent (the original .safetensors works too).
Qwen2.5-VL-7B text encoder — prompt → 3584-dim conditioning, with an optional mmproj vision tower for image-grounded edits.

Place the companions next to the DiT GGUF, or point at them with TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ (CLI: --qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj). The pipeline VAE-encodes the reference, builds the conditioning, runs a FlowMatch-Euler true-CFG denoise loop (reference-latent concatenation), then VAE-decodes back to pixels. The whole 60-block DiT forward is CUDA-graph-captured and flash attention is on by default; the target area auto-clamps to the device VRAM budget unless you pin a width/height.

An optional Lightning distillation LoRA (--qwen-image-lora / TS_QWEN_IMAGE_LORA, .safetensors) is merged into the DiT at load time — it auto-derives the reduced denoise step count (e.g. 4 or 8) and switches CFG to 1.0, skipping the negative pass. The denoise loop is further accelerated by a whole-step DiT cache (EasyCache-style; TS_QWEN_DIT_CACHE_MODE = easycache / fbc) that skips DiT steps when the output is predictable from the previous step.

Run it from the CLI (--image + --prompt), the Web UI image-edit flow (with live denoising previews), or in-process from C# via QwenImageModel.EditImage(). Full details are in the repository's docs/models/qwenimage.md card.

Thinking / reasoning mode

Thinking-capable models (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H, DeepSeek V4) produce structured chain-of-thought before the final answer. The thinking content is separated from the visible response so the client can show or hide it.

Qwen 3 / Qwen 3.5/3.6 / Nemotron-H — <think>…</think> tags.
Gemma 4 — <|channel>thought …<channel|> tags.
GPT OSS — Harmony format: <|channel|>analysis for thinking, <|channel|>final for the answer.
DeepSeek V4 — <think>…</think> tags; the chat template closes the block immediately unless thinking is requested, so reasoning is opt-in.

Enable it via --think (CLI), "think": true (Ollama API / Web UI), or the thinking toggle in the browser. Responses expose the reasoning separately — e.g. message.thinking in the Ollama chat response.

Tool calling / function calling

Models can invoke user-defined tools and participate in multi-turn tool-call conversations. Define tools as JSON and pass them via --tools (CLI) or the tools parameter (API). Each architecture uses its own wire format, but the output parser extracts calls into structured tool_calls regardless:

Qwen 3 / Nemotron-H — <tool_call>{"name": …, "arguments": {…}}</tool_call>
Qwen 3.5 / 3.6 — the same <tool_call> block with an XML body: <function=NAME><parameter=key>value</parameter></function> (the JSON form is still accepted).
Gemma 4 — <|tool_call>call:function_name{args}<tool_call|>
GPT OSS (Harmony) — tools declared as a TypeScript namespace; calls emitted on the commentary channel.
DeepSeek V4 — DSML markup: the system prompt teaches the syntax and carries one JSON schema per function, and the model answers with <｜DSML｜tool_calls><｜DSML｜invoke name="NAME"><｜DSML｜parameter name="key" string="true|false">value</｜DSML｜parameter>….

See Tool calling over HTTP for a complete request/response example and the continuation loop.

← Compute Backends Next: Command Line →