Server & Web UI

TensorSharp.Server.Host (built on the TensorSharp.Server library) is an ASP.NET Core app that hosts a single GGUF model and exposes a browser chat UI plus Ollama- and OpenAI-compatible REST APIs on the same port. The continuous-batching engine handles concurrency.

Embedding service

Start with --model encoder.gguf --embeddings to host Snowflake Arctic Embed L v2.0 or MiniLM. POST /v1/embeddings and POST /api/embed return normalized vectors on pure C# cpu or native ggml_cpu / ggml_metal / ggml_cuda. Downloads, APIs, truncation, and retrieval quality →

Jev typed decisions

Host a DiffusionGemma GGUF (for example with config/jev-diffusiongemma-q4.json) and POST /v1/systemone answers Jev-compatible typed-decision requests: the body carries state and typed questions (noul, choice, score), and the reply carries probabilities, choices and expected scores read after one denoising step. model may name the loaded model or the aliases jev-latest / jev-preview, which are not separate checkpoints. Requests are JSON only; with the vision tower loaded through --mmproj, a request may also carry up to 8 inline images (base64 or data: URLs) in its images array. TS_JEV_MAX_BODY_MB (default 8, 1–64), TS_JEV_MAX_CANVAS (default 64, 8–4096) and TS_JEV_MAX_PENDING (default 32 admitted requests, 1–1024) bound the work. Fields, limits and status codes →

Start the server

Quick start in ~30 seconds (Gemma 4 E4B)

Install the .NET 10 SDK for your platform, Git, CMake, and curl, then paste this from a terminal. Copying and running the commands takes about 30 seconds; the 7.48 GiB model download and the first restore/build take longer and depend on your connection and machine. It hosts the repository's benchmark-verified Gemma 4 E4B Q8_0 from the recommended public ggml-org artifact on the native GGML bridge. This block is for Linux + NVIDIA (the CUDA build also needs the CUDA Toolkit):

git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
dotnet run --project TensorSharp.Server.Host -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan; with no supported GPU, drop the assignment and use ggml_cpu. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf is in the same repository. Text needs no projector; image, video, or audio also requires the matching mmproj-gemma-4-E4B-it-Q8_0.gguf passed with --mmproj. See Getting Started for Windows PowerShell and full platform syntax.

In a second terminal, verify the OpenAI-compatible endpoint:

curl -s http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Reply with one short hello."}],"max_tokens":32}'

The API base is http://localhost:5000. When the bundled wwwroot/ is present and enabled, GET / serves its index.html chat UI. Use GET /health for the stable plain-text liveness check; only a headless deployment with no enabled Web UI falls back to that health response at the bare / route.

Already-built source tree

Run these commands from the repository root after building. They invoke TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll; that output directory also contains the copied native libraries and wwwroot/. The Releases page also provides attached, self-contained CLI and server archives for supported platforms.

# Apple Silicon / Metal
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_metal

# NVIDIA / GGML CUDA
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_cuda

# AMD, Intel, or NVIDIA / Vulkan; inspect device indices first
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --list-gpus
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_vulkan --gpu-device 1

# Multimodal: the projector is always explicit
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --mmproj ./models/mmproj.gguf --backend ggml_cuda

Server-wide default sampling

Defaults fill in any field a request omits. Out of the box they match Ollama: temperature 0.8, top-k 40, top-p 0.9, min-p 0, repeat penalty 1.1 over the last 64 tokens, presence/frequency 0, seed -1. A parameter you set explicitly also overrides the value a client sends for it — many chat clients (VS Code Copilot Chat among them) hardcode temperature/top_p into every request. Add --sampling-precedence request to hand that control back to clients.

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_metal \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.1 \
    --presence-penalty 0.0 --frequency-penalty 0.0 --seed 42 \
    --stop "</s>" --stop "<|endoftext|>"

Web UI features

Open http://localhost:5000/. The browser interface supports:

• Multi-turn chat conversations with streaming token generation (SSE).
• Answers rendered from Markdown: tables, code blocks, lists, headings and links. The page builds them as elements rather than parsing the model's text as HTML, so a reply cannot inject markup or script; a link keeps its target only for http(s), mailto and the server's own file downloads.
• A stats line under each answer: the tokens the turn generated (reasoning and tool calls included), its wall-clock seconds, the decode speed, and how much of the prompt was served from the KV cache.
• Per-tab chat sessions — each tab owns tracked conversation history; request KV blocks and prefix reuse are owned by the inference engine.
• Image, video, audio, PDF, and text/code uploads for multimodal inference (500 MB per file by default; --upload-max-mb changes the per-file cap, and POST /api/upload's request-body limit follows it upward — see the upload options below).
• PDF documents: born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models (TS_PDF_MAX_PAGES caps the pages read). The final rendered prompt is checked against the model's actual context window.
• Thinking / reasoning mode toggle and tool calling with function definitions.
• A live activity row for agentic turns; while wait_agent runs it offers a collapsible wait_agent · N sub-agents list with one card per sub-agent (task, status, current tool, result), updated in place.
• Message editing and deletion with regeneration from any point in the conversation.
• DiffusionGemma denoising previews when a diffusion-gemma GGUF is hosted (the whole assistant message is replaced on each step, then finalized).
• Qwen-Image-2.1 flow when a qwen_image DiT is hosted: a prompt without an attachment generates an image, and attaching one or more images edits them; live denoising previews (up to 8 frames) refresh in place until the final PNG appears with a download link. The browser sends no size, so the output uses --width / --height when both are set, otherwise the model's automatic size.
• Video flow when a video-generation model is hosted — MiniMax-H3 or Wan: type the prompt, attach whatever conditioning the loaded checkpoint advertises, and the browser streams per-step denoise progress over SSE until the MP4 appears with a download link. On MiniMax-H3 a 32 kHz stereo soundtrack comes back beside it, generated in the same pass. Everything numeric still comes from the startup flags — see Video generation below.
• Free scrolling — read earlier replies while new tokens stream; auto-scroll resumes at the bottom.

Selected-area image edits: attach photos, choose Select area on the editing target, save the selection and describe the change. Qwen-Image 2.1 keeps the original dimensions and protects pixels outside the selection. The result offers Compare original and Edit again; other attached photos remain references. Selection editor and mask API →

Agentic work and code execution

On tool-capable model families, the server runs the loaded model through an in-process generate → tool → generate loop. Agent Skills may be discovered and read without executing scripts; --skills-allow-exec separately opts the operator into bundled skill scripts. --code-exec opts into the host-handled read_file, write_file, apply_patch, and sandboxed shell tools. Web UI chat sessions retain a private workspace across rounds, while stateless API requests receive an isolated request workspace that is removed afterward.

The same model can also delegate: on /v1/chat/completions, /v1/responses, /api/chat/ollama and the Web UI's /api/chat, sub-agent delegation is on by default for families that render tool declarations and parse tool calls (not Mistral 3, Hunyuan Dense or DiffusionGemma). The model decides whether to call spawn_agent, wait_agent, send_input, close_agent and list_agents; it needs neither skills nor --code-exec. Each child is a separate conversation on the same model with its own copy of the KV state, and explorer and reviewer children are read-only. --no-multi-agent turns it off, the --agents-* options below bound it, and a request can opt out with "multi_agent": false. No latency or quality measurements are published for it. → Sub-agents

Video generation: what the UI sets, and what only the API sets

Host a video-generation model — MiniMax-H3 or Wan — and the Web UI turns into a video generator. The endpoints gate on the IVideoGenerationModel seam rather than on an architecture string, so anything else answers 400 with The loaded model is not a video-generation model. It is worth knowing exactly how little of the request the browser controls, because a video job can run for minutes or for hours and the browser gives you no way to change that mid-flight.

# MiniMax-H3: the clip and its 32 kHz stereo soundtrack, denoised together in one latent
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model ./models/minimax_h3_fl2va_pruned-Q4_K.gguf --backend ggml_cuda \
    --video-width 640 --video-height 384 --video-steps 20 --video-frames 22

# Same geometry, the other checkpoint: identity/appearance references into a new scene
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model ./models/minimax_h3_ref2va_pruned-Q4_K.gguf --backend ggml_cuda \
    --video-width 640 --video-height 384 --video-steps 20 --video-frames 22

FL2VA and Ref2VA are two checkpoints, not two settings: which one is loaded decides whether attached images are keyframes or references, and it is what /api/models reports to the browser. config/minimax-h3-fl2va.json and config/minimax-h3-ref2va.json carry the same geometry and download the denoiser, the shared Qwen3-VL-32B text encoder and both VAEs on first run — but not the tokenizer, which is not a flag: put vocab.json and merges.txt from the upstream MiniMaxAI/MiniMax-H3 processor/ folder beside the encoder, or point TS_VIDEO_TOKENIZER at the folder holding them.

What the process decides at startup, and how:

The Web-UI-shaped endpoint takes the fields under their JSON names; the same body works on /api/video-generate (one MP4 back) and /api/video-generate/stream (SSE denoise progress). Generations are serialized process-wide, so one job runs at a time:

curl -s http://localhost:5000/api/video-generate \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "a red fox trotting through falling snow, cinematic",
        "width": 640, "height": 384,
        "frames": 22, "fps": 24,
        "steps": 8, "cfg": 1.0, "seed": 42,
        "videoMode": "t2v",
        "generateAudio": true
      }'

The reply is { ok, url, audioUrl, width, height, frames, fps, seed, codec, elapsedSeconds }; audioUrl points at the sidecar WAV and is null when the model produced no track. Wan takes the same body with its own knobs instead — "flowShift": 5.0, "sampler": "unipc", "negativePrompt", "cfgCacheStride": 2 — none of which MiniMax-H3 uses, because it runs guidance-free at "cfg": 1.0.

An "image" field carries a base64 conditioning frame (a data:…;base64, prefix is accepted); "imagePath", "endImage" and every reference* entry are the Web UI's form and must name files already returned by /api/upload — anything resolving outside the upload directory is rejected. The OpenAI-shaped POST /v1/videos/generations takes the same parameters but spells the size as "size": "832x480" and the negative prompt as "negative_prompt". Full request/response shapes are on the HTTP API page.

Configuration file (--config)

Instead of a long command line, pass a JSON file with --config. The CLI reads the same format. Command-line options always win—when the command line sets a single-valued option itself, that option's entry in the file is dropped before it is resolved (not emitted, its variables not substituted, its download never attempted), so one file can be reused across hosts while you override just what differs. Each override prints one [config] --X is set on the command line, so the value from '<file>' is ignored … line to stderr. This holds for --gpu-device and --kv-cache-dtype too. Repeat --config to layer files (a later file's entry replaces an earlier one's the same way). The repeatable options — --stop, --skills-dir, --skill, --lora, --lora-scale, --lora-config, --image and the --ref-* inputs — keep the file's values and add the command line's after them. Comments and trailing commas are allowed.

# Read all options from a file; override just the backend for this host
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json --backend ggml_cpu

Keys are the same long option names below (with or without the leading --). A string/number becomes --key value, true becomes the bare switch --key, and an array becomes a repeated flag (e.g. "stop": ["</s>", "<|eot|>"]). false and null emit nothing, so to turn a default-on option off name its negation ("no-continuous-batching": true). lora-scale and lora-config cannot be arrays, and a removed option is refused as a key. Because the server rejects unknown options, a CLI-only key (such as interactive or input) stops it at startup; the penalty window is repeat-last-n on both hosts.

Variables. Define shared values once under "variables" (alias "vars") and reference them with ${name}, or ${name:-fallback} for a default, in any string value (an undefined name falls back to an environment variable of the same name). Declare as many roots as you need — models in different folders each get their own.

Auto-download. Any file option can be an object with a local path and one or more urls (or a single url). A relative path resolves against the config file's folder. If path is missing it downloads from the first working URL (mirrors tried in order), saves it there, and reuses it next time; progress prints to stderr, and an optional sha256 verifies a fresh download (an existing file is used as-is). A download entry whose option the command line (or a later file) also sets is dropped before it is resolved, so --mmproj none on the command line both disables a configured projector and skips its download.

{
  "variables": { "modelRoot": "C:/models", "repo": "https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/bfc15c382204943c3a8fff0c750b94ae2364d7a3" },
  "backend": "ggml_cuda",
  "max-tokens": 4096,
  "continuous-batching": true,
  "stop": ["</s>", "<|eot|>"],
  "model":  { "path": "${modelRoot}/gemma-4-E4B-it-Q8_0.gguf", "urls": [ "${repo}/gemma-4-E4B-it-Q8_0.gguf" ] },
  "mmproj": { "path": "${modelRoot}/gemma-4-E4B-mmproj-F16.gguf", "urls": [ "${repo}/mmproj-F16.gguf" ] }
}

Ready-to-use examples live in the repository's config/ folder (cli-basic.json, server-basic.json, variables.json, auto-download.json, qwen-image-2.1.json) — each uses real, public, ungated URLs, so it works on a fresh machine. See config/README.md for the full reference.

Server options

Options are case-insensitive and also accept the --option=value form. Unlike the CLI, the server refuses to start on an unknown option or a stray positional argument, and suggests the closest known option when one is within two edits.

Environment variables

The server listens on http://0.0.0.0:5000 by default; --port, --host and --urls (or the PORT / HOST / ASPNETCORE_URLS environment variables) move it, and the Docker Space images set PORT=7860.

Continuous-batching tunables

The scheduler / engine knobs are read at process start. Set them via the environment (or the --continuous-batching flags, which translate to them).

The full environment-variable surface (MLX tunables, MTP knobs, diffusion) is on the API Reference page and the Advanced page.