Server & Web UI

TensorSharp.Server is an ASP.NET Core app that hosts a single GGUF model and exposes a browser chat UI plus Ollama- and OpenAI-compatible REST APIs on the same port. The continuous-batching engine handles concurrency.

Start the server

Quick start in ~30 seconds (Gemma 4 E4B)

Install the .NET 10 SDK for your platform, Git, CMake, and curl, then paste this from a terminal. Copying and running the commands takes about 30 seconds; the 7.48 GiB model download and the first restore/build take longer and depend on your connection and machine. It hosts the repository's benchmark-verified Gemma 4 E4B Q8_0 from the recommended public ggml-org artifact on the native GGML bridge. This block is for Linux + NVIDIA (the CUDA build also needs the CUDA Toolkit):

git clone https://github.com/zhongkaifu/TensorSharp.git
cd TensorSharp
TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
dotnet run --project TensorSharp.Server -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan; with no supported GPU, drop the assignment and use ggml_cpu. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf is in the same repository. Text needs no projector; image, video, or audio also requires the matching mmproj-gemma-4-E4B-it-Q8_0.gguf passed with --mmproj. See Getting Started for Windows PowerShell and full platform syntax.

In a second terminal, verify the OpenAI-compatible endpoint:

curl -s http://localhost:5000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gemma-4-E4B-it-Q8_0.gguf","messages":[{"role":"user","content":"Reply with one short hello."}],"max_tokens":32}'

The API base is http://localhost:5000. Open the bundled chat UI at http://localhost:5000/index.html; GET / is the liveness endpoint and returns "TensorSharp.Server is running".

Already-built source tree

Run these commands from the repository root after building. They invoke TensorSharp.Server/bin/TensorSharp.Server.dll; that output directory also contains the copied native libraries and wwwroot/. The current v3.0.5.0 GitHub release has no attached binary archives.

# Apple Silicon / Metal
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal

# NVIDIA / GGML CUDA
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_cuda

# AMD, Intel, or NVIDIA / Vulkan; inspect device indices first
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --list-gpus
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_vulkan --gpu-device 1

# Multimodal: the projector is always explicit
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --mmproj ./models/mmproj.gguf --backend ggml_cuda

Server-wide default sampling

Defaults fill in any field a request omits. Out of the box they match Ollama: temperature 0.8, top-k 40, top-p 0.9, min-p 0, repeat penalty 1.1, presence/frequency 0, seed -1. A parameter you set explicitly also overrides the value a client sends for it — many chat clients (VS Code Copilot Chat among them) hardcode temperature/top_p into every request. Add --sampling-precedence request to hand that control back to clients.

dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model ./models/model.gguf --backend ggml_metal \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.1 \
    --presence-penalty 0.0 --frequency-penalty 0.0 --seed 42 \
    --stop "</s>" --stop "<|endoftext|>"

Web UI features

Open http://localhost:5000/index.html. The browser interface supports:

• Multi-turn chat conversations with streaming token generation (SSE).
• Per-tab chat sessions — each tab owns tracked conversation history; request KV blocks and prefix reuse are owned by the inference engine.
• Image, video, audio, PDF, and text/code uploads for multimodal inference (up to 500 MB).
• PDF documents: born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models (TS_PDF_MAX_PAGES caps the pages read). The final rendered prompt is checked against the model's actual context window.
• Thinking / reasoning mode toggle and tool calling with function definitions.
• Message editing and deletion with regeneration from any point in the conversation.
• DiffusionGemma denoising previews when a diffusion-gemma GGUF is hosted (the whole assistant message is replaced on each step, then finalized).
• Qwen-Image-Edit flow when a qwen_image DiT is hosted: attach an image, type the edit instruction, and temporary edited images (live denoising previews, up to 8 frames) refresh in place until the final PNG appears with a download link.
• Free scrolling — read earlier replies while new tokens stream; auto-scroll resumes at the bottom.

Configuration file (--config)

Instead of a long command line, pass a JSON file with --config. The CLI reads the same format. Command-line options always win—file values are applied first, then anything you also pass on the command line overrides them, so one file can be reused across hosts while you override just what differs. Repeat --config to layer files (later files win). Comments and trailing commas are allowed.

# Read all options from a file; override just the backend for this host
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basic.json
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --config config/server-basic.json --backend ggml_cpu

Keys are the same long option names below (with or without the leading --). A string/number becomes --key value, true becomes the bare switch --key, and an array becomes a repeated flag (e.g. "stop": ["</s>", "<|eot|>"]).

Variables. Define shared values once under "variables" and reference them with ${name} in any string value (an undefined name falls back to an environment variable of the same name). Declare as many roots as you need — models in different folders each get their own.

Auto-download. Any file option can be an object with a local path and one or more urls. If path is missing it downloads from the first working URL (mirrors tried in order), saves it there, and reuses it next time; progress prints to stderr, and an optional sha256 verifies the file.

{
  "variables": { "modelRoot": "C:/models", "repo": "https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF/resolve/main" },
  "backend": "ggml_cuda",
  "max-tokens": 4096,
  "continuous-batching": true,
  "stop": ["</s>", "<|eot|>"],
  "model":  { "path": "${modelRoot}/gemma-4-E4B-it-Q8_0.gguf", "urls": [ "${repo}/gemma-4-E4B-it-Q8_0.gguf" ] },
  "mmproj": { "path": "${modelRoot}/gemma-4-E4B-mmproj-F16.gguf", "urls": [ "${repo}/mmproj-F16.gguf" ] }
}

Ready-to-use examples live in the repository's config/ folder (cli-basic.json, server-basic.json, variables.json, auto-download.json, qwen-image-edit.json) — each uses real, public, ungated URLs, so it works on a fresh machine. See config/README.md for the full reference.

Server options

Environment variables

The current binary listens on a fixed http://0.0.0.0:5000; the Docker Space images patch that constant to 7860 at build time.

Continuous-batching tunables

The scheduler / engine knobs are read at process start. Set them via the environment (or the --continuous-batching flags, which translate to them).

The full environment-variable surface (MLX tunables, MTP knobs, diffusion) is on the API Reference page and the Advanced page.