Server & Web UI

TensorSharp.Server.Host (built on the TensorSharp.Server library) is an ASP.NET Core app that hosts a single GGUF model and exposes a browser chat UI plus Ollama- and OpenAI-compatible REST APIs on the same port. The continuous-batching engine handles concurrency.

Start with the chat recipe below, or go straight to embeddings, Jev, agents or video for another model or protocol. The reference tables retain every option and expand when needed.

1. Start a local server

Quick start (Gemma 4 E4B)

Install the .NET 10 SDK for your platform, Git, CMake, and curl, then paste this from a terminal. Allow time for the 7.48 GiB model download and first restore/build; duration depends on your connection and machine. It hosts the example Gemma 4 E4B Q8_0 from the recommended public ggml-org artifact on the native GGML bridge. This block is for Linux + NVIDIA (the CUDA build also needs the CUDA Toolkit):

πŸ”

The standalone server has no built-in API-key authentication. The commands below use HTTP (no TLS) and explicitly bind 127.0.0.1 for local access. Omitting --host uses the default all-interface binding, 0.0.0.0. For remote access, use a host firewall and an authenticated HTTPS reverse proxy; do not expose the server port directly to an untrusted network.

git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
dotnet run --project TensorSharp.Server.Host -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --host 127.0.0.1

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan; with no supported GPU, drop the assignment and use ggml_cpu. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf is in the same repository. Text needs no projector; image, video, or audio also requires the matching mmproj-gemma-4-E4B-it-Q8_0.gguf passed with --mmproj. See Getting Started for Windows PowerShell and full platform syntax.

After the startup log shows the listen address, check liveness and the loaded model from a second terminal, then send one chat request:

curl -sS http://127.0.0.1:5000/health
curl -sS http://127.0.0.1:5000/v1/models
curl -s http://127.0.0.1:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Reply with one short hello."}],"max_tokens":32}'

The API base is http://127.0.0.1:5000. When the bundled wwwroot/ is present and enabled, GET / serves its index.html chat UI. GET /health always returns the liveness response "TensorSharp.Server is running"; it does not check model readiness. The bare / falls back to that response when the Web UI is disabled or index.html is missing.

πŸ“Œ

--model is required for inference. The server hosts exactly the startup GGUF and optional, explicitly supplied --mmproj; it does not scan a model directory or auto-detect a projector. /api/models/load can only re-load that same startup pair, optionally on another supported backend. A model-less process cannot choose a GGUF at runtime. The default listen address is http://0.0.0.0:5000; --port, --host and --urls (env PORT / HOST / ASPNETCORE_URLS) move it.

Already-built source tree

Already built the source, or need another backend? Show launch examples

Run these commands from the repository root after building. They invoke TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll; that output directory also contains the copied native libraries and wwwroot/. The Releases page also provides attached, self-contained CLI and server archives for supported platforms.

# Apple Silicon / Metal
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_metal --host 127.0.0.1

# NVIDIA / GGML CUDA
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_cuda --host 127.0.0.1

# AMD, Intel, or NVIDIA / Vulkan; inspect device indices first
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --list-gpus
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_vulkan --gpu-device 1 --host 127.0.0.1

# Multimodal: the projector is always explicit
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --mmproj ./models/mmproj.gguf --backend ggml_cuda --host 127.0.0.1

2. Use the browser or an API

Your goalNext stepCheck
Chat in a browserOpen http://127.0.0.1:5000/, then see UI features.The enabled Web UI needs its deployed wwwroot/index.html.
Use an OpenAI-compatible clientUse API base http://127.0.0.1:5000/v1; see Chat Completions and Responses.Tools, reasoning and media depend on the model and endpoint.
Use an Ollama-compatible clientFollow the Ollama request examples./api/chat/ollama is the compatibility route; /api/chat uses the Web UI SSE protocol.

Web UI features

Open http://127.0.0.1:5000/. The browser interface supports:

Browser features, uploads and generation workflows
The TensorSharp Web UI: Qwen3.8 27B compared two mortgages by writing and running a Python script, answered with a table and a recommendation, and offered the script and the CSV as downloads
Qwen3.8 27B (UD-Q4_K_XL) on ggml_metal, started with --code-exec, on an Apple M5 Pro. The model wrote a Python script, ran it in the macOS sandbox, checked that both schedules end at a $0 balance, answered with a table, and offered the script and the 540-row CSV as downloads.

Selected-area image edits: attach photos, choose Select area on the editing target, save the selection and describe the change. Qwen-Image 2.1 keeps the original dimensions and protects pixels outside the selection. The result offers Compare original and Edit again; other attached photos remain references. Selection editor and mask API β†’

Steer a reply during generation

Type and send a new message while a reply is generating. The host cancels and drains the old generation, retains its produced reply and starts the new turn; the separate Stop button still cancels. This requires a build after v2026.10.10.

Personal knowledge bases

Set TS_KNOWLEDGE_DIRECTORY before startup to enable the store and /api/knowledge management API. Supported chat requests explicitly pass knowledgeCollections; missing or empty selection retrieves nothing. The standalone server Chat page has no TensorAgent collection-management panel, so use the API/SDK. TensorAgent separately supplies collection, folder and chat-import UI. Configuration, supported routes and limits β†’

3. Configure and operate it

Keep a working local launch, then make configuration, access and storage explicit in your deployment. The options, environment variables and scheduler reference retain the full details.

Configuration file (--config)

Config precedence, variables and automatic downloads

Instead of a long command line, pass a JSON file with --config. The CLI reads the same format. Command-line options always winβ€”when the command line sets a single-valued option itself, that option's entry in the file is dropped before it is resolved (not emitted, its variables not substituted, its download never attempted), so one file can be reused across hosts while you override just what differs. Each override prints one [config] --X is set on the command line, so the value from '<file>' is ignored … line to stderr. This holds for --gpu-device and --kv-cache-dtype too. Repeat --config to layer files (a later file's entry replaces an earlier one's the same way). For --skills-dir, the winning source replaces the complete root list; repeated values within that source survive. Accumulating options β€” --stop, --skill, --lora, --lora-scale, --lora-config, --image and the --ref-* inputs β€” keep the file's values and add the command line's after them. Comments and trailing commas are allowed.

# Read options from a file; bind loopback and override the backend for this host
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json --host 127.0.0.1
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/server-basic.json --backend ggml_cpu --host 127.0.0.1

Keys are the same long option names below (with or without the leading --). A string/number becomes --key value, true becomes the bare switch --key, and an array becomes a repeated flag (e.g. "stop": ["</s>", "<|eot|>"]). false and null emit nothing, so to turn a default-on option off name its negation ("no-continuous-batching": true). lora-scale and lora-config cannot be arrays, and a removed option is refused as a key. Because the server rejects unknown options, a CLI-only key (such as interactive or input) stops it at startup; the penalty window is repeat-last-n on both hosts.

Variables. Define shared values once under "variables" (alias "vars") and reference them with ${name}, or ${name:-fallback} for a default, in any string value (an undefined name falls back to an environment variable of the same name). Declare as many roots as you need β€” models in different folders each get their own.

Auto-download. Any file option can be an object with a local path and one or more urls (or a single url). A relative path resolves against the config file's folder. If path is missing it downloads from the first working URL (mirrors tried in order), saves it there, and reuses it next time; progress prints to stderr, and an optional sha256 verifies a fresh download (an existing file is used as-is). A download entry whose option the command line (or a later file) also sets is dropped before it is resolved, so --mmproj none on the command line both disables a configured projector and skips its download.

{
  "variables": { "modelRoot": "C:/models", "repo": "https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/bfc15c382204943c3a8fff0c750b94ae2364d7a3" },
  "backend": "ggml_cuda",
  "max-tokens": 4096,
  "continuous-batching": true,
  "stop": ["</s>", "<|eot|>"],
  "model":  { "path": "${modelRoot}/gemma-4-E4B-it-Q8_0.gguf", "urls": [ "${repo}/gemma-4-E4B-it-Q8_0.gguf" ] },
  "mmproj": { "path": "${modelRoot}/gemma-4-E4B-mmproj-F16.gguf", "urls": [ "${repo}/mmproj-F16.gguf" ] }
}

Ready-to-use examples live in the repository's config/ folder (cli-basic.json, server-basic.json, variables.json, auto-download.json, qwen-image-2.1.json) β€” each uses real, public, ungated URLs, so it works on a fresh machine. See config/README.md for the full reference.

Server-wide default sampling

Sampling defaults and request precedence

Defaults fill in any field a request omits. Out of the box they match Ollama: temperature 0.8, top-k 40, top-p 0.9, min-p 0, repeat penalty 1.1 over the last 64 tokens, presence/frequency 0, seed -1. A parameter you set explicitly also overrides the value a client sends for it β€” many chat clients (VS Code Copilot Chat among them) hardcode temperature/top_p into every request. Add --sampling-precedence request to hand that control back to clients.

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model ./models/model.gguf --backend ggml_metal --host 127.0.0.1 \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.1 \
    --presence-penalty 0.0 --frequency-penalty 0.0 --seed 42 \
    --stop "</s>" --stop "<|endoftext|>"

Uploads and retention

BoundaryDefault and control
One file500 MB by default; --upload-max-mb controls multipart uploads and decoded base64 attachments.
HTTP request bodyPOST /api/upload follows the per-file cap and never drops below 500 MB. Other routes stay at 500 MB; base64 expands by about a third, limiting an inline file to roughly 375 MB before other JSON fields.
Total budget--upload-quota-mb is off by default; generated images and videos count, and excess requests are rejected before model work.
Retention--upload-ttl-hours is off by default because sessions reference attachment paths. TENSORSHARP_UPLOAD_DIR moves the default uploads/ directory beside the binary.

Choose a service mode for your task

Select the model and protocol for the task. Embeddings and Jev already have detailed guides; the host-specific limits remain below, with video and tool details available on demand.

Embedding service guide Β· Jev API Β· Agents and tools Β· Video models and downloads

Embedding service

Encoders, media projectors and backend scope

Host Snowflake, MiniLM or EmbeddingGemma 2 with --model encoder.gguf --embeddings. POST /v1/embeddings and POST /api/embed return normalized vectors; EmbeddingGemma 2 with --mmproj also accepts image/audio/video and mixed content sequences. Backend support and measured scope depend on the model. Recipes, APIs and validation β†’

Jev typed decisions

Jev requests, aliases and workload limits

Host a DiffusionGemma GGUF (for example with config/jev-diffusiongemma-q4.json) and POST /v1/systemone answers Jev-compatible typed-decision requests: the body carries state and typed questions (noul, choice, score), and the reply carries probabilities, choices and expected scores read after one denoising step. model may name the loaded model or the aliases jev-latest / jev-preview, which are not separate checkpoints. Requests are JSON only; with the vision tower loaded through --mmproj, a request may also carry up to 8 inline images (base64 or data: URLs) in its images array. TS_JEV_MAX_BODY_MB (default 8, 1–64), TS_JEV_MAX_CANVAS (default 64, 8–4096) and TS_JEV_MAX_PENDING (default 32 admitted requests, 1–1024) bound the work. Fields, limits and status codes β†’

Agentic work and code execution

Tools, sub-agents and sandbox boundaries

On tool-capable model families, the server runs the loaded model through an in-process generate β†’ tool β†’ generate loop. Agent Skills may be discovered and read without executing scripts; --skills-allow-exec separately opts the operator into bundled skill scripts. --code-exec opts into the host-handled read_file, write_file, apply_patch, and sandboxed shell tools. Web UI chat sessions retain a private workspace across rounds, while stateless API requests receive an isolated request workspace that is removed afterward.

The same model can also delegate: on /v1/chat/completions, /v1/responses, /api/chat/ollama and the Web UI's /api/chat, sub-agent delegation is on by default for families that render tool declarations and parse tool calls (not Mistral 3, Hunyuan Dense or DiffusionGemma). The model decides whether to call spawn_agent, wait_agent, send_input, close_agent and list_agents; it needs neither skills nor --code-exec. Each child is a separate conversation on the same model with its own copy of the KV state, and explorer and reviewer children are read-only. --no-multi-agent turns it off, the --agents-* options below bound it, and a request can opt out with "multi_agent": false. No latency or quality measurements are published for it. β†’ Sub-agents

πŸ”’

Executable tools are off until the operator enables them, but there is no per-command approval prompt after that decision. The default sandbox mode is required and fails closed: macOS uses Seatbelt and Linux requires bubblewrap 0.12.0 or newer. Windows Job Objects bound descendants but do not confine filesystem or network access, so skill scripts require an explicit less-safe sandbox mode and model-authored commands require --code-exec-unconfined. Host tool calls anywhere in one request's sub-agent tree run one at a time, and a worker child gets the parent's mutable tools only with --agents-allow-worker-tools. See Agentic Work, Skills & Sandboxing for the complete flags, model-family limits, workspace lifecycle, and security boundaries.

Video generation: what the UI sets, and what only the API sets

Model recipes, startup flags and per-request video fields

LTX 2.5: use --config config/ltx-2.5-distilled.json; NVIDIA uses cuda/ggml_cuda, Apple uses mlx/ggml_metal (the MLX executor), and CPU aliases select direct CPU. Defaults are 1024Γ—1024Γ—17 at 24 fps, dimensions divisible by 64 and frames 8k+1; eight half-size steps plus three refinement steps, CFG 1. Five files, audio, conditioning and limits β†’

Host a video-generation model β€” LTX 2.5, MiniMax-H3 or Wan β€” and the Web UI turns into a video generator. The endpoints gate on the IVideoGenerationModel seam rather than on an architecture string, so anything else answers 400 with The loaded model is not a video-generation model. It is worth knowing exactly how little of the request the browser controls, because a video job can run for minutes or for hours and the browser gives you no way to change that mid-flight.

# MiniMax-H3: the clip and its 32 kHz stereo soundtrack, denoised together in one latent.
# No size flags: text-to-video runs at 640Γ—384 and an uploaded keyframe sizes the canvas
# by MiniMax's 768p recipe; --video-width/--video-height would pin one size for every mode
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --host 127.0.0.1 \
    --model ./models/minimax_h3_fl2va_pruned-Q4_K.gguf --backend ggml_cuda \
    --video-steps 20 --video-frames 22

# The other checkpoint: identity/appearance references into a new scene, at 640Γ—384
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --host 127.0.0.1 \
    --model ./models/minimax_h3_ref2va_pruned-Q4_K.gguf --backend ggml_cuda \
    --video-steps 20 --video-frames 22

FL2VA and Ref2VA are separate checkpoints for keyframe and reference modes. The single config/minimax-h3-fl2va.json downloads both, plus the shared Qwen3-VL-32B encoder, both VAEs and the pinned tokenizer files. The model switches denoisers per request; /api/models advertises the available modes. The config leaves output size to the selected workflow. Manual tokenizer installs can set TS_VIDEO_TOKENIZER.

To serve FL2VA on LightX2V's 8- or 4-step Turbo recipe, add its plug-in config. Every clip then samples the plug-in's schedule, so leave --video-steps unset (or set it to the plug-in's count) and send requests without steps, as the Web UI does: a contradicting --video-steps stops startup before anything downloads, and a contradicting request gets a 400 (a {done, error} event on the stream route). The weights (1.38 GB) download on first use; the 4-step recipe is inferred. β†’ Turbo plug-ins

dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --config config/minimax-h3-fl2va.json --host 127.0.0.1 \
    --lora config/lora/minimax-h3-fl2v-turbo-8step.json
⏱️

The browser sends the prompt plus whatever conditioning the loaded model advertises β€” a first frame, a last frame, or up to maxReferenceImages reference stills alongside any attached clips and soundtracks β€” shaped from the video capability object that GET /api/models returns rather than guessed from an architecture name. Frame count, fps, resolution, denoise steps, seed, flow shift, sampler, negative prompt and the guidance cache are not exposed in the UI: they come from the startup flags or the model's own recipe. Guidance has no startup flag at all β€” the server takes no --cfg β€” so a request is the only place to set it, and MiniMax-H3 refuses anything above 1.0 because it ships CFG-distilled. To set any of these per request, call POST /api/video-generate or /v1/videos/generations directly.

What the process decides at startup, and how:

What decides itSettingNotes
--model (fixed for the process)Which family, and how many denoise passesThe single biggest cost factor. MiniMax-H3 is CFG-distilled, so every step is one guidance-free pass β€” 20 steps by default; its step-distilled schedule comes from a plug-in rather than another checkpoint (next row). A base Wan2.2-TI2V-5B runs the official 50 steps Γ— 2 CFG passes = 100 DiT passes; a step-distilled Turbo / Lightning / FastWan checkpoint is detected from the file name and runs 4 guidance-free passes. On an M5 Pro at 1088Γ—832 Γ— 121 frames that is β‰ˆ3 h 30 m versus 17 m 30 s for the identical request. β†’ H3 measurements Β· Wan measurements Β· downloads
--lora <plug-in> (fixed for the process)MiniMax-H3's step-distilled scheduleThe LightX2V Turbo plug-ins for the FL2VA checkpoint, config/lora/minimax-h3-fl2v-turbo-8step.json and -4step.json, run every clip in 8 or 4 guidance-free passes instead of 20, on a locked recipe (the 4-step recipe is inferred). Ref2VA refuses them. No Turbo clip timings are recorded yet.
--video-width N / --video-height NDefault output sizeThe main quality lever on the server, because the Web UI sends no size of its own: without these every clip is generated at the model's default. For MiniMax-H3, leave both unset: text-to-video and references then run at 640Γ—384, and image-to-video and first/last-frame take the canvas from the first keyframe by MiniMax's 768p recipe (1152Γ—768 for a 3:2 photo, 1344Γ—768 for 16:9). Those modes keep the keyframe's framing, and at 640Γ—384 a wide shot's faces blur; the recipe size takes several times longer. Setting both pins one size for every mode, keyframes included, and turns off Studio's β€œFrom the first frame” size; when only one of the two is given H3 takes the other from the conditioning image's aspect ratio. Aliases: --width / --height.
--video-steps NDefault denoise stepsThe quality/time trade-off after resolution. MiniMax-H3 defaults to 20; 16–24 is visibly cleaner, past ~30 gains little, and fewer steps come from a Turbo plug-in (row above). A request's steps overrides it; under a Turbo plug-in both must be unset or equal the plug-in's count, and anything else is refused (at startup, or with a 400).
--video-mode <mode>Default conditioning modePins t2v / i2v / fl2v / ref for requests that omit videoMode. Omit it and each request infers its own mode from what it supplies, which is usually what you want; pin it for a deployment that only offers one.
--video-frames NDefault frame countA server-wide default, not a cap: a request that carries frames overrides it. Snapped to the model's temporal grid β€” 4k+1 for Wan, 17k+5 for MiniMax-H3. With the flag omitted the model recipe applies β€” 49 frames for Wan2.2-TI2V, 33 for the other Wan checkpoints, 22 for MiniMax-H3.
--fps NDefault playback rateAlso a default, overridden independently by a request's fps. Omitted, the recipe gives 24 fps for Wan2.2-TI2V and 16 otherwise; MiniMax-H3 is trained at a fixed 24 fps and overrides any other value. FPS changes playback, not the amount of work.
--audio-vae <path>Whether a soundtrack comes backOnly for models that generate audio jointly with the video (minimax_h3_audio_vae_fp32.safetensors). Without it MiniMax-H3 still runs and produces video, just silent β€” and supportsAudio on /api/models reports false.
The request (API only)Everything elsewidth, height, frames, steps, cfg, cfg2, seed, fps, flowShift, sampler, negativePrompt, cfgCacheStride, videoMode, generateAudio, endImage, referenceImages, referenceVideos, referenceAudios, referenceVideoAudios.

The Web-UI-shaped endpoint takes the fields under their JSON names; the same body works on /api/video-generate (one MP4 back) and /api/video-generate/stream (SSE denoise progress). Generations are serialized process-wide, so one job runs at a time:

curl -s http://127.0.0.1:5000/api/video-generate \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "a red fox trotting through falling snow, cinematic",
        "width": 640, "height": 384,
        "frames": 22, "fps": 24,
        "steps": 8, "cfg": 1.0, "seed": 42,
        "videoMode": "t2v",
        "generateAudio": true
      }'

The reply is { ok, url, audioUrl, width, height, frames, fps, seed, codec, elapsedSeconds }; audioUrl points at the sidecar WAV and is null when the model produced no track. Wan takes the same body with its own knobs instead β€” "flowShift": 5.0, "sampler": "unipc", "negativePrompt", "cfgCacheStride": 2 β€” none of which MiniMax-H3 needs, because it runs guidance-free at "cfg": 1.0 (under a Turbo plug-in, a flowShift or sampler other than the plug-in's own is refused).

An "image" field carries a base64 conditioning frame (a data:…;base64, prefix is accepted); "imagePath", "endImage" and every reference* entry are the Web UI's form and must name files already returned by /api/upload β€” anything resolving outside the upload directory is rejected. The OpenAI-shaped POST /v1/videos/generations takes the same parameters but spells the size as "size": "832x480" and the negative prompt as "negative_prompt". Full request/response shapes are on the HTTP API page.

πŸ’‘

Before starting a long job, size it. Cost is dominated by DiT tokens (latent_frames Γ— (h/2) Γ— (w/2)) and self-attention is O(tokensΒ²), so frames and frame area matter far more than steps. On MiniMax-H3 the order is resolution, then frames, then steps β€” it is already guidance-free, and its one distillation lever is a Turbo plug-in (FL2VA only), which samples 8 or 4 passes instead of 20. On Wan: use a step-distilled checkpoint, cut frames, cut frame area (but not below ~0.3 MP), cut steps on base checkpoints, then "cfgCacheStride": 2 or 3 for 1.30Γ— / 1.43Γ—. 480p (β‰ˆ0.4 MP) is a resolution Wan is trained at, so it is a real output mode rather than a degraded one.

Full reference

Expand the table you need; each summary can be focused and toggled with the keyboard.

Server options

All server options and behavior

Options are case-insensitive and also accept the --option=value form. Unlike the CLI, the server refuses to start on an unknown option or a stray positional argument, and suggests the closest known option when one is within two edits.

OptionDescription
--model <path>GGUF file to host (required for inference).
--mmproj <path|none>Explicit multimodal projector: an mmproj GGUF, or β€” for the Gemma 4 family, DiffusionGemma included β€” a HuggingFace .safetensors shard holding the vision tower (diffusiongemma-26B-A4B-it is published without an mmproj). A bare filename resolves next to the model; pass none to disable it. Requires --model; there is no automatic projector scan.
--embeddings Β· --embedding-threads <N> Β· --embedding-context-size <N>Host a GGUF embedding encoder instead of a chat model (see Embedding service); requires --model. Snowflake/MiniLM support cpu, ggml_cpu, ggml_metal and ggml_cuda; EmbeddingGemma 2 additionally implements direct cuda/mlx and accepts media through a matching --mmproj. Vulkan embeddings are unimplemented; availability does not establish all-backend media quality or performance parity. The generation routes (chat, Responses, Ollama, /v1/systemone, and /api/image-generate, /api/image-edit, /api/video-generate with their /stream forms) answer 400 This server hosts an embedding model. Use /v1/embeddings or /api/embed.; an explicit --backend this machine does not have exits 2 with error: model load refused: Backend 'X' is not supported on this machine. The other two set the encoder's CPU threads and its maximum tokens per input, and require --embeddings.
--backend <type>Default backend: cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, ggml_vulkan (default: ggml_metal on macOS, ggml_cpu elsewhere; env BACKEND).
--tp <N>Tensor-parallel degree, for sharding weights within layers only (default 1; env TENSORSHARP_TP_DEGREE). Requires a supported model and GPU backend. Mutually exclusive with --layer-split; unsupported requests fail at startup.
--layer-split <N>Local GPU count for whole-layer placement on supported architectures; env TENSORSHARP_LAYER_SPLIT_DEGREE. With --tp T above 1 it forms a hybrid placement (P stages of T GPUs each, one process) on architectures that support it; multi-node layer split takes --tp-node-id/--tp-peers without --tp.
--tp-node-id <N> / --tp-peers <list>Join a multi-node TP cluster: this node's 0-based ID and the shared, identically-ordered host:port list of every node. The server can only be node 0 β€” the driver that owns sampling and serves HTTP; other nodes run TensorSharp.Cli workers.
--gpu-device <N>Vulkan device index for the ggml_vulkan backend on multi-GPU hosts (default: 0; env TS_GGML_VULKAN_DEVICE).
--list-gpusList the Vulkan devices ggml-vulkan can see (index + adapter name) and exit.
--port <N> Β· --host <address> Β· --urls <urls>Listen port (default 5000; env PORT), bind address (default 0.0.0.0; env HOST), or full semicolon-separated listen URLs for cases the other two cannot express (falls back to ASPNETCORE_URLS; --port / --host win when both are given).
--no-webuiDo not serve the bundled Web UI; GET / answers the liveness response. Every API endpoint, /uploads included, stays up. Env: TS_NO_WEBUI (any value but 0).
--helpPrint the full parameter reference and exit; it is also shown when no arguments are supplied. Inference always requires a startup --model.
--config <path>Read options from a JSON config file (command-line options override it). Supports ${variables} and auto-downloading models via { "path": ..., "urls": [...] }. Repeatable.
--max-tokens <N>Generation limit for every endpoint (Web UI, Ollama and OpenAI alike): fills in when a request omits it, and caps a request that asks for more (default: 20000, which only fills in).
--temperature / --top-k / --top-p / --min-pSampling values (defaults: 0.8 / 40 / 0.9 / 0).
--repeat-penalty / --presence-penalty / --frequency-penalty / --seedPenalties and seed (defaults: 1.1 / 0 / 0 / -1).
--repeat-last-n <N>How many recent tokens the penalties consider (default 64; 0 disables, -1 uses all; env TENSORSHARP_REPEAT_LAST_N). Requests set it as repeat_last_n. The CLI uses the same spelling; the former CLI-only --penalty-last-n is removed and refused on both hosts.
--stop <string>Stop sequence (repeatable). Merged with a per-request stop list under config precedence; replaced by it under request.
--sampling-precedence <config|request>Who wins when a request also carries a sampling parameter you configured above: config (default) keeps your value, request lets the client's win. Parameters you did not configure always come from the request. Env: TENSORSHARP_SAMPLING_PRECEDENCE.
--continuous-batching / --no-continuous-batchingEnable (default) or disable iteration-level paged batching.
--no-prefix-cacheTurn off runtime prefix reuse (the Radix prefix cache, on by default), the startup preparation of the shared system/tool prompt, and its persistence between launches; it also sets TS_SCHED_PREFIX_CACHE=0. With it on, the server prefills the shared prompt once at startup and saves the result, so the first message of a process costs the same as any other (measured 21.8 s β†’ 0.7 s on an agent configuration); the price is that the first launch after a prompt, skills or model change opens its port only once that preparation finishes. Checkpoints live in prefix-cache/<model>/ beside the binary, two files per model; TENSORSHARP_PREFIX_CACHE_DIR moves the root.
--spec / --no-specEnable / disable speculative decoding (default off) β€” the explicit opt-in for drafters embedded in the trunk checkpoint, since loading them pages extra weights into VRAM, and the switch --spec-type ngram needs. With the default auto type it engages only on a checkpoint that carries a drafter β€” for Qwen 3.6 that means a GGUF that retains the NextN block (e.g. the -MTP- repos); base-repo GGUFs strip it and silently fall back to standard decode. Qwen 3.8 27B, GLM-5.2 and GLM-5.3 already carry theirs in the trunk GGUF, so nothing extra is downloaded, but the flag still has to be on the command line before the model loads: it is what pages glm-dsa's NextN layer into VRAM (~3 GiB at GLM-5.2's IQ2_XXS), which also leaves less room for the context. It is refused under --tp N>1, where the draft block would have to borrow a column-parallel trunk LM head, so on those two GLM releases speculation engages on single-device or explicit --layer-split N placement (no active tensor parallelism). Nemotron-H refuses every speculator: under --spec it serves plain decoding and says so once. β†’ MTP
--spec-type <name>Speculation algorithm: auto (default) uses whatever drafter the checkpoint carries; draft-head and block pin one explicitly; ngram needs no trained weights and drafts by suffix match over the context. It still needs a trunk that verifies speculatively, so it does not run on GPT-OSS, Mistral 3 or Hunyuan Dense; DeepSeek V4 / V4.1 and Muse-Glimmer verify only through their own block drafter, so there it runs only while that drafter is loaded with --draft-model; and Nemotron-H refuses every speculator. --spec-type only chooses the algorithm β€” turn n-gram on with --spec --spec-type ngram. Given without --spec or --draft-model (and with TS_SPEC unset), it, --spec-draft and --spec-pmin leave speculation off, and the server logs a startup WARNING that says so. Env: TS_SPEC_TYPE.
--spec-draft <N>Max tokens drafted per speculative step (default 8; a block drafter defaults to β€” and clamps the value to β€” its trained block size).
--spec-pmin <f>Minimum draft confidence to keep a token; 0 means never gate. The default depends on the algorithm: 0.15 for a per-token draft head, 0.35 for a block drafter (where the gate is the cumulative prefix probability, so the same number is far stricter), and 0 for ngram (where it scales the required match length instead).
--draft-model <path>Drafter GGUF for every speculator that ships as its own file β€” DeepSeek V4's DSpark support module, the DFlash / DFlash2 drafters for Muse-Glimmer and Qwen 3.8, Gemma 4's gemma4-assistant draft head, and Qwen 3.8 Flash Next's shared MTP head (GGML backends). The file's own general.architecture decides how it loads, and naming the file enables speculation by itself β€” no --spec needed (an explicit --no-spec vetoes it). Qwen 3.6, Qwen 3.8 27B, GLM-5.2 and GLM-5.3 embed their NextN draft in the trunk GGUF and need no draft file β€” GLM-5.3's is complete at blk.78, and --spec is what enables it. If an explicitly requested draft cannot be activated on the startup model, startup fails fast (exit code 2) β€” a DFlash / DFlash2 drafter under --tp N > 1, or any drafter on Nemotron-H, for example. DeepSeek V4's DSpark engages for solo sequences on the cuda and ggml_cuda backends; DeepSeek V4.1 accepts a deepseek41-dspark drafter on ggml_cuda / ggml_cpu only, an experimental path; initial text/image HTTP probes with trained weights passed using two-GPU layer split on ggml_cuda; broad quality and throughput remain unqualified. Verification draws every row with the request's own sampler, so it composes with any sampling settings. Env: TS_SPEC_DRAFT_MODEL. β†’ DSpark
--prefill-chunk-size <N>Maximum prefill tokens per scheduler step (sets TS_SCHED_PREFILL_CHUNK).
--kv-cache-dtype <type>KV cache precision: f32, f16, q8_0, or q4_0 (default: auto β€” the backend/model pick; env KV_CACHE_DTYPE; q4_0 targets very long 128K–256K contexts).
--n-cpu-moe <N|all> / -ncmoe Β· --cpu-moe / -cmoe Β· --cpu-moe-threads <N>MoE CPU offload, as on the CLI: keep the routed experts of the first N layers (or all of them) in system RAM and multiply them on the CPU. Default 0 on every architecture. The host thread count defaults to 1 with 2 or fewer usable CPUs, all but one with up to 8, and half above that, capped at 64; DeepSeek V4 / V4.1 and GLM-5.x on their native executors use every usable CPU once experts are offloaded (GLM-5.x also on a GPU-less run). Env: TS_N_CPU_MOE, TS_CPU_MOE, TS_CPU_MOE_THREADS.
--redis-url <url>Back the OpenAI Responses API store with Redis (TS_RESPONSES_STORE_REDIS_URL) instead of process memory. An already-set TS_RESPONSES_STORE_REDIS_URL is left as it is. --help lists it under "Responses API store", and startup logs Redis configured via --redis-url: the Responses API store uses <url>.
--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path>Override the resolved Qwen-Image-2.1 companions (dedicated 2.1 VAE / Qwen3-VL-8B text encoder / Qwen3-VL-8B mmproj).
--qwen-image-variant base|turboWhich Qwen-Image-2.1 checkpoint the DiT GGUF holds (the GGUFs carry no metadata). turbo samples Qwen-Image-2.1-Turbo's published 8-step schedule at CFG 1, refuses other step counts and refuses step-distillation --lora plug-ins. Without it, a file name with the word turbo is assumed to be Turbo and the load says so. Another model loads without it, with a warning. Env: TS_QWEN_IMAGE_VARIANT.
--lora <path> / --lora-scale <f> / --lora-config <path>LoRA plug-ins for the hosted Qwen-Image-2.1 or MiniMax-H3 (FL2VA) model, with the CLI's spelling and binding rules (repeat --lora to stack Qwen-Image-2.1 plug-ins; scale and config bind to the preceding --lora). Checked at startup, where a plug-in the startup model does not take is refused before the plug-in downloads (a model a --config still has to download is checked right after that download); a model loaded later runs without plug-ins it does not take and logs a warning naming them. Applied to every image request or clip; per-request LoRA selection is not implemented. On Qwen-Image-2.1 a request's steps / cfg still override a plug-in's sampling recipe; a MiniMax-H3 Turbo plug-in locks --video-steps and a request's steps, flowShift and sampler to its recipe. β†’ Qwen-Image-2.1 plug-ins Β· MiniMax-H3 Turbo
--width <px> / --height <px>Default Qwen-Image-2.1 output size for image requests that name neither a size nor an area β€” which includes every Web UI request (sets TS_QWEN_IMAGE_WIDTH / TS_QWEN_IMAGE_HEIGHT); a request with its own size or target area keeps its own geometry. The default needs both: a value off the 32-pixel grid is rounded down to a multiple of 32 (never below 32) with a one-time warning, and with only one side (or an unparsable or negative value) the default is ignored with a one-time warning and the automatic size (a 2048Γ—2048 area) stays. A Qwen-Image server warns at startup in either case; nothing is refused. The same two flags are also aliases of --video-width / --video-height. LTX 2.5 defaults to 1024Γ—1024 and requires dimensions divisible by 64.
--qwen-image-lora Β· --offload-cpuRemoved and rejected at startup, including as --config keys. Both served only the earlier Qwen-Image-Edit pipeline. --qwen-image-lora is replaced by --lora; --offload-cpu has no replacement, because Qwen-Image-2.1 keeps its DiT weights resident.
--penalty-last-nRemoved and rejected at startup, including as a --config key (exit code 1). It was the CLI-only name of the repeat-penalty window: use --repeat-last-n <N>.
--paged-kv* Β· --paged-bench* Β· --paged-batchingRemoved and rejected at startup, including as --config keys (exit code 1). The standalone paged-KV store they configured is gone: prompt reuse across requests is the radix prefix cache (--no-prefix-cache), and its TS_KV_* variables are refused too. --paged-batching / --no-paged-batching were second spellings of --continuous-batching / --no-continuous-batching.
--video-width <px> / --video-height <px>Default output size when a request omits width/height. The main video quality lever, because the Web UI sends no size of its own. Leave both unset for MiniMax-H3: text-to-video and references run at 640Γ—384, and i2v/fl2v take the first keyframe's 768p recipe size (1152Γ—768 for a 3:2 photo), which a wide shot's faces need. Setting both pins one size for every mode, keyframes included; when only one of the two is given H3 takes the other from the conditioning image's aspect ratio. Aliases: --width / --height (which also set the Qwen-Image-2.1 default size). LTX 2.5 defaults to 1024Γ—1024 and requires dimensions divisible by 64.
--video-steps <N>Default denoising steps when a request omits steps. MiniMax-H3 defaults to 20; 16–24 is visibly cleaner, past ~30 gains little. For fewer MiniMax-H3 steps, load a Turbo plug-in with --lora instead: it samples exactly the 8 or 4 steps it was distilled for, so leave this unset or set it to that count; any other value is refused at startup, and a request's own steps likewise. LTX 2.5 accepts the fixed eight-step first pass and automatically adds three refinement steps.
--video-mode <mode>Default conditioning mode when a request omits videoMode: t2v, i2v, fl2v or ref. Omit it and each request infers its own mode from what it supplies, which is usually what you want.
--video-frames <N>Default output frame count when a request omits frames, snapped to the model's temporal grid (4k+1 for Wan, 17k+5 for MiniMax-H3). Model default: 33, or 49 for Wan2.2-TI2V, 22 for MiniMax-H3. A request value overrides it β€” this is a default, not a cap. LTX 2.5 defaults to 17 frames/24 fps and requires the 8k+1 frame grid.
--fps <N>Default MP4 playback rate when a request omits fps. Model default: 16, or 24 for Wan2.2-TI2V. Models trained at a fixed rate (MiniMax-H3, 24 fps) override any other value. FPS changes playback rate, not generation work.
--video-vae <path>Video VAE β€” wan_2.1_vae.safetensors, Wan2.2_VAE.safetensors for TI2V-5B, minimax_h3_video_vae_fp16.safetensors for MiniMax-H3, or the preset convolutional video VAE for LTX 2.5. Default: same-directory scan next to the DiT, VAE/ subfolders included. Env: TS_VIDEO_VAE.
--video-text-encoder <path>Text encoder β€” UMT5-XXL GGUF for Wan, Qwen3-VL-32B GGUF for MiniMax-H3, or the dedicated Gemma 4 projection safetensors for LTX 2.5. Default: same-directory scan (env TS_VIDEO_TEXT_ENCODER). --video-dit2 / TS_VIDEO_DIT2 names Wan 2.2 A14B's second high/low-noise expert when the pair is not co-located.
--audio-vae <path>Audio VAE for models that generate an audio track jointly with the video (minimax_h3_audio_vae_fp32.safetensors). Companion requirements are model-specific; use the complete LTX preset. Env: TS_VIDEO_AUDIO_VAE.
--video-upscaler <path>x2 latent upscaler for two-stage video models: LTX 2.5 generates at half the output size, upscales the latent with ltx-2.5-latent-spatial-upscaler-x2-bf16-1.0.safetensors, then refines at full size. Default: latent_upscale_models/ beside the model. Env: TS_VIDEO_UPSCALER.
--video-diffusion-vae <path>Opt-in LTX 2.5 diffusion video decoder (ltx-2.5-video-vae-bf16.safetensors): cleaner, with less grain and flicker, but softer than the default convolutional --video-vae. MLX and CPU only; CUDA refuses it. Default: off. Env: TS_VIDEO_DIFFUSION_VAE.
--upload-max-mb <N>Per-file cap in MB on client-originated writes: multipart /api/upload files and base64 attachments decoded out of chat requests (default 500; env TS_UPLOAD_MAX_MB). The request-body limit of POST /api/upload follows the cap and never drops below 500 MB, so a larger cap lets a larger file in there, to be referenced by path afterwards. Every other route keeps the 500 MB request-body limit, and a base64 file grows by about a third, so an attachment inside a JSON request tops out near 375 MB whatever the cap.
--upload-quota-mb <N>Total budget in MB for the upload directory, generated images and videos included; a request that would exceed it is rejected before any model work runs. Default: off; env TS_UPLOAD_QUOTA_MB.
--upload-ttl-hours <N>Delete upload-directory files older than this many hours (fractions allowed). Default: off, because chat sessions reference attachments by path; enable it when untrusted clients can reach the server. Env: TS_UPLOAD_TTL_HOURS. The directory is uploads/ beside the binary unless TENSORSHARP_UPLOAD_DIR moves it.
--skills-dir Β· --skill Β· --list-skills Β· --no-skills Β· --skills-no-discovery Β· --skills-allow-exec Β· --skills-sandbox Β· --skills-allow-network Β· --skills-max-roundsAgent Skills, with the same spellings, defaults and environment variables as the CLI (default roots: every .agents/skills from the working directory up to the Git root, then skills/ beside the binary). On the server, skills/ beside the binary β€” where POST /api/skills installs uploads β€” is always scanned first, even when roots are given, so it wins a name clash; explicit roots and TS_SKILLS_DIR replace only the .agents/skills defaults. Also on the server, --skill selects a skill for every request unless the request names its own, uploads land in skills/ beside the binary, and --no-skills leaves /v1/skills and /api/skills unmapped and answers a request that names skills with HTTP 400.
--code-exec and its --code-exec-* optionsThe built-in code tools and their install, network, timeout, output, shell, temperature and confinement settings, with the same spellings and defaults as the CLI. --code-exec-languages is removed and rejected at startup.
--no-multi-agentTurn off sub-agent delegation, which is on by default for tool-capable families (see above). Env: TS_NO_MULTI_AGENT set to any value other than 0.
--agents-max-concurrent Β· --agents-max-count Β· --agents-max-depthActive descendants across a request's tree, root excluded (default 3, range 1–32) Β· children created per request tree (8, 1–128) Β· delegation depth below the root (2, 1–8).
--agents-max-rounds Β· --agents-max-generations Β· --agents-timeout Β· --agents-max-result-charsTool rounds per child turn (default 8, range 1–64) Β· child generations shared by the whole request (48, 1–1024) Β· seconds per child run (180, 1–3600) Β· characters per child report (8000, 256–64000).
--agents-allow-worker-toolsLet worker children use the parent's enabled mutable tools. Off by default; explorer and reviewer children are always read-only.
βœ…

Per-request fields (temperature, top_p, seed, stop, …) fill in every parameter you did not configure here. For one you did configure, your value wins unless the server runs with --sampling-precedence request. The chat.start log line prints the sampler each request actually ran with.

Environment variables

Environment variables, defaults and state directories
VariableDescription
BACKENDDefault backend when --backend is not passed (default: ggml_metal on macOS, ggml_cpu elsewhere).
MAX_TOKENSDefault max generation length (default: 20000).
MAX_CONTEXTHard context limit. Left unset, an advertised context is a ceiling: after the weights load the engine asks the devices how much VRAM is actually free and sizes the context to fit the caches plus one full n_ubatch graph, logging its pick (GLM-5.2 on 3Γ— RTX PRO 6000: 342,272 tokens on the layer split, 91,136 with --tp 3, 646,400 with --n-cpu-moe 30). Set it and the value is honoured if it fits and refused with the numbers if it does not, rather than quietly shrunk.
TS_PDF_MAX_PAGESCap on PDF pages read during upload β€” text extraction and page-image rendering (default: 0 = all pages; also honored by the CLI's --pdf).
VIDEO_SAMPLE_FPS / VIDEO_MAX_FRAMESFrames sampled per second of video / optional upper bound on extracted frames.
TENSORSHARP_TEMPERATURE, …_TOP_K, …_TOP_P, …_MIN_PDefault sampling values when neither the flag nor the request body sets one.
TENSORSHARP_REPEAT_PENALTY, …_REPEAT_LAST_N, …_PRESENCE_PENALTY, …_FREQUENCY_PENALTY, …_SEEDDefault penalties, penalty window and seed.
TENSORSHARP_LOG_LEVEL / …_LOG_DIR / …_LOG_FILELogger level, directory, and file toggle β€” the only way to configure server logging, which has no --log-* flags (the CLI honors them too, as defaults for its own flags).
TENSORSHARP_PREFIX_CACHE_DIRRoot for the server's persisted prefix checkpoints (default prefix-cache/ beside the binary); each model gets its own subdirectory. See --no-prefix-cache.
TENSORSHARP_UPLOAD_DIRUpload directory (default uploads/ beside the binary), served under /uploads/.
DIFFUSION_STEPS / DIFFUSION_MAX_BATCHDiffusionGemma denoising steps per block / max concurrent diffusion requests batched.
TENSORSHARP_TP_DEGREELocal tensor-parallel degree, equivalent to --tp N; only shards weights inside layers.
TENSORSHARP_LAYER_SPLIT_DEGREELocal GPU count for whole-layer placement, equivalent to --layer-split N; mutually exclusive with tensor-parallel settings.
TENSORSHARP_TP_DEVICESGPU ordinals the TP ranks map to, e.g. 0,2 (default 0..tp-1). GGML backends.
TENSORSHARP_TP_NODE_ID / TENSORSHARP_TP_PEERSExtend tensor parallelism across machines: this node's 0-based ID and the shared comma-separated host:port list of every node (also --tp-node-id / --tp-peers). Set both, or neither. The server can only be node 0 β€” the driver that serves HTTP; other nodes run TensorSharp.Cli workers.
TS_RESPONSES_STORE_REDIS_URLBack the OpenAI Responses API store with Redis instead of process memory. Server flag: --redis-url.

The server listens on http://0.0.0.0:5000 by default; --port, --host and --urls (or the PORT / HOST / ASPNETCORE_URLS environment variables) move it, and the Docker Space images set PORT=7860.

Continuous-batching tunables

Continuous batching and scheduler parameters

The scheduler / engine knobs are read at process start. Set them via the environment (or the --continuous-batching flags, which translate to them).

VariableDescription
TS_SCHED_DISABLE_BATCHED1 forces per-sequence KV-swap even when a model supports batching (= --no-continuous-batching).
TS_SCHED_MAX_BATCHED_TOKENSPer-step token budget (default 4096).
TS_SCHED_MAX_RUNNING_SEQSMaximum in-flight sequences (default 16).
TS_SCHED_PREFILL_CHUNKPer-request prefill cap while a decode is active (default 256); prefill-only steps fill the complete token budget.
TS_SCHED_SOLO_PREFILL_CHUNKPrefill chunk size when at most one sequence is in the system β€” caps all solo prefill chunks (default 8192, bounded by TS_SCHED_MAX_BATCHED_TOKENS). Left unset, a model that sizes its own lone-prompt steps does so instead: Muse Spark with pipelined prefill takes 8192-token steps, joining a remainder of up to 2048 tokens to the step before, and may exceed that budget. Setting it turns the model's steps off; the engine log says which applies.
TS_SCHED_DECODE_QUANTUMDecode tokens before a sequence switch (default 256 = block size).
TS_SCHED_NUM_BLOCKS / TS_SCHED_BLOCK_SIZEPhysical blocks in the engine pool (default 256) / tokens per block (default 16 for a family that reuses only whole pages, otherwise 256).
TS_SCHED_PREFIX_CACHE0 disables all runtime prefix reuse across requests (= what --no-prefix-cache sets).
TS_BATCHED_FUSED_DECODEBatched fused decode is enabled by default on native slot paths (DeepSeek V4, GLM 5.x): one graph, one token per sequence, weights read once for the whole batch β€” 1.81Γ— aggregate decode at 4 concurrent requests on GLM-5.2. Set 0 to use serial fused decode. Batching changes GEMM shapes, and a 2-bit MoE can amplify that into different expert picks.

The full environment-variable surface (MLX tunables, MTP knobs, diffusion) is on the API Reference page and the Advanced page.