API Reference
Every flag, variable, endpoint, and public type in one place. Type in the box to filter all tables below instantly — or press / for wiki-wide search.
· Matching rows are highlighted live across all sections.
CLI flags — TensorSharp.Cli
| Flag | Description |
|---|---|
--model <path> | Path to a GGUF model file (required). |
--input <path> | Text file containing the user prompt. |
--input-jsonl <path> | JSONL file with batch requests (one JSON per line). |
--multi-turn-jsonl <path> | JSONL for multi-turn chat simulation with KV-cache reuse. |
--output <path> | Write generated text to this file. |
--image / --video / --audio <path> | Media for vision / video / audio inference. |
--pdf <path> | PDF document input (one-shot): born-digital PDFs are inlined as text; scanned PDFs become page images for a vision model (page cap: TS_PDF_MAX_PAGES). |
--mmproj <path> | Multimodal projector GGUF. Pass it explicitly; CLI auto-detection recognizes only a few legacy companion filenames. |
--max-tokens <N> | Maximum tokens to generate (default 100). |
--backend <type> | cpu, cuda, mlx, ggml_cpu, ggml_metal, ggml_cuda, ggml_vulkan. |
--gpu-device <N> | Vulkan device index for ggml_vulkan on multi-GPU hosts (default 0; env TS_GGML_VULKAN_DEVICE). |
--list-gpus | List visible Vulkan devices (index + adapter name) and exit. |
--kv-cache-dtype <type> | KV cache precision: f32, f16, q8_0, q4_0 (default: auto per backend/model; q4_0 ~1/7 of f32, for very long 128K–256K contexts; the quantized tiers need the native GGML flash-attention path). |
--interactive / -i | Start the interactive REPL. |
--system <text> / --system-file <path> | Seed the system prompt. |
--think | Enable thinking / reasoning mode. |
--tools <path> | JSON file with tool / function definitions. |
--draft-model <path> | Speculative-decoding drafter GGUF for architectures whose drafter ships as its own file (DeepSeek V4's DSpark). Needs --backend cuda or ggml_cuda and a pure-argmax sampler. Env TS_DSV4_DSPARK. |
--spec-draft-n-max <N> / --spec-draft-conf-min <p> | Tokens drafted per speculative block (default: the drafter's block size, 5) and the minimum cumulative acceptance probability to keep a drafted position (default 0.35). |
--temperature / --top-k / --top-p / --min-p | Sampling controls. |
--repeat-penalty / --presence-penalty / --frequency-penalty | Penalties (1.0 / 0 = off). |
--seed <N> / --stop <string> | Random seed (-1 = random) / stop sequence (repeatable). |
--dump-prompt | Render prompt + tokenization and exit. |
--diffusion-steps / --diffusion-seed / --diffusion-blocks <N> | DiffusionGemma generation controls. |
--prompt <text> | Qwen-Image-Edit edit instruction (with --image; the edited PNG goes to --output). |
--cfg <F> | Qwen-Image-Edit true-CFG guidance scale (auto: 2.5, or 1.0 with a Lightning LoRA; <= 1 disables the negative pass). |
--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path> | Override the resolved Qwen-Image-Edit companion GGUFs (VAE / Qwen2.5-VL text encoder / mmproj). |
--qwen-image-lora <path> | Lightning distillation LoRA (.safetensors) merged into the DiT; auto-derives the step count (e.g. 4 or 8) and switches CFG to 1.0. |
--paged-kv-quant-bits <0|2|4|8> | TurboQuant paged-KV block compression (0 = off). |
--benchmark / --bench-prefill / --bench-decode / --bench-runs | Synthetic throughput benchmark. |
--bench-kvcache / --bench-kv-turns <N> | Multi-turn KV-cache reuse benchmark. |
--warmup-runs <N> | Throw-away forward passes before timing (default 0). |
--test / --test-templates <dir> | Built-in tokenizer/template tests; validate templates against GGUF Jinja2. |
--tp <N> | Tensor parallelism degree — split the model across N GPUs in one process (default 1; requires --backend cuda, ggml_cuda, or ggml_vulkan). Also accepted by TensorSharp.Server. |
--tp-node-id <N> / --tp-peers <list> | Multi-node distributed TP: this node's 0-based ID and the shared comma-separated host:port peer list. |
--log-level / --log-dir / --log-file / --log-console | Logger level, directory, and file/console toggles. |
Server flags — TensorSharp.Server
| Flag | Description |
|---|---|
--model <path> | GGUF file to host. Required at startup; a model-less process cannot select a GGUF through /api/models/load. |
--mmproj <path> | Explicit multimodal projector GGUF; none to disable. The server does not auto-detect it. |
--backend <type> | Compute backend; defaults to ggml_metal on macOS and ggml_cpu elsewhere. |
--gpu-device <N> / --list-gpus | Vulkan device selection for ggml_vulkan / list visible Vulkan devices and exit. |
--help | Print the parameter reference and exit (also shown when started with no arguments). |
--max-tokens <N> | Generation limit for every endpoint: fills in when a request omits it and caps a larger request (default 20000, which only fills in). |
--temperature / --top-k / --top-p / --min-p | Default sampling values. |
--repeat-penalty / --presence-penalty / --frequency-penalty / --seed | Default penalties and seed. |
--stop <string> | Stop sequence (repeatable); merged with a per-request list under config precedence, replaced by it under request. |
--sampling-precedence <config|request> | Whether configured sampling parameters outrank the ones a request sends (default config). Parameters left unconfigured always come from the request. |
--continuous-batching / --no-continuous-batching | Enable (default) / disable iteration-level paged batching. Alias --paged-batching. |
--mtp-spec / --no-mtp-spec | Enable / disable NextN/MTP speculative decoding (default off). |
--mtp-draft <N> | Max tokens drafted per speculative step (default 8). |
--mtp-pmin <f> | Minimum draft confidence to keep a token (default: per drafter kind — 0.75 per-token, 0.35 cumulative for a block drafter). |
--mtp-draft-model <path> | Separate MTP draft GGUF (Gemma 4 gemma4-assistant). |
--draft-model <path> | Block drafter GGUF that must be resident before the layer split (DeepSeek V4's DSpark). Needs --mtp-spec; solo sequences on cuda / ggml_cuda. |
--spec-draft-n-max <N> / --spec-draft-conf-min <p> | Block-drafter size cap and cumulative acceptance-probability gate (default 0.35). |
--prefill-chunk-size <N> | Maximum prefill tokens per scheduler step (sets TS_SCHED_PREFILL_CHUNK). |
--kv-cache-dtype <type> | KV cache precision: f32, f16, q8_0, q4_0 (default: auto per backend/model; env KV_CACHE_DTYPE). |
--qwen-image-vae / --qwen-image-vl / --qwen-image-mmproj <path> | Override the resolved Qwen-Image-Edit companion GGUFs. |
--qwen-image-lora <path> | Qwen-Image-Edit Lightning LoRA merged into the DiT (fewer denoise steps, CFG 1.0). |
--paged-kv* / --paged-kv-quant-bits | Legacy standalone paged-KV flags (engine now owns KV state). |
Environment variables
| Variable | Description |
|---|---|
BACKEND | Default backend (ggml_metal on macOS, ggml_cpu elsewhere). |
MAX_TOKENS | Default max generation length (20000). |
TS_PDF_MAX_PAGES | Cap on PDF pages read (upload and CLI --pdf); 0 = all pages (default). |
VIDEO_SAMPLE_FPS / VIDEO_MAX_FRAMES | Video frame sampling rate / cap. |
TENSORSHARP_TEMPERATURE / _TOP_K / _TOP_P / _MIN_P | Default sampling values. |
TENSORSHARP_REPEAT_PENALTY / _PRESENCE_PENALTY / _FREQUENCY_PENALTY / _SEED | Default penalties and seed. |
TENSORSHARP_SAMPLING_PRECEDENCE | config (default) or request: whether the values above outrank the ones a client sends (= --sampling-precedence). |
TENSORSHARP_LOG_LEVEL / _LOG_DIR / _LOG_FILE | Logging level, directory, file toggle (CLI + server). |
DIFFUSION_STEPS / DIFFUSION_MAX_BATCH | DiffusionGemma steps per block / max batched requests. |
KV_CACHE_DTYPE | KV cache precision (CLI + server): f32, f16, q8_0, q4_0; default auto (= --kv-cache-dtype). |
TS_SCHED_DISABLE_BATCHED | 1 forces per-sequence KV-swap (= --no-continuous-batching). |
TS_SCHED_MAX_BATCHED_TOKENS | Per-step token budget (4096). |
TS_SCHED_MAX_RUNNING_SEQS | Max in-flight sequences (16). |
TS_SCHED_PREFILL_CHUNK | Max prefill tokens per step (1024). |
TS_SCHED_SOLO_PREFILL_CHUNK | Prefill chunk for a solo / uncontended request (8192). |
TS_SCHED_DECODE_QUANTUM | Decode tokens before a sequence switch (256 = block size). |
TS_SCHED_NUM_BLOCKS / TS_SCHED_BLOCK_SIZE | Engine block-pool size (256) / tokens per block (256). |
TS_SCHED_PREFIX_CACHE | 0 disables block-hash prefix sharing. |
TS_<FAMILY>_BATCHED | 0 forces a family onto the per-sequence path (e.g. TS_GEMMA4_BATCHED, TS_QWEN35_BATCHED). |
TS_MTP_SPEC / TS_MTP_DRAFT / TS_MTP_PMIN / TS_MTP_DRAFT_MODEL | MTP speculative-decoding knobs (mirror the --mtp-* flags). |
TS_DSV4_DSPARK / TS_DSV4_NGPU / TS_DSV4_UBATCH / TS_DSV4_PERF | DeepSeek V4: DSpark drafter path (mirrors --draft-model), GPU count for the layer split, prefill micro-batch, and throughput/stage logging. |
TS_GMTP_NO_FUSED / TS_GMTP_NO_FAST_ROLLBACK / TS_GMTP_BATCHED_TRUNK | Gemma 4 MTP draft-path A/B switches. |
TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ | Qwen-Image-Edit companion paths (VAE / Qwen2.5-VL text encoder / mmproj). |
TS_QWEN_IMAGE_LORA / TS_QWEN_IMAGE_LORA_SCALE | Lightning LoRA .safetensors merged into the DiT / scale override. |
TS_QWEN_DIT_CACHE / TS_QWEN_DIT_CACHE_MODE | Whole-DiT-step cache: 0 disables / mode easycache, fbc, both, off. |
TENSORSHARP_TP_DEGREE | Local tensor-parallel degree — GPUs to split the model across (= --tp on the CLI and the server). |
TENSORSHARP_TP_DEVICES | GPU ordinals the TP ranks map to, e.g. 0,2 (default 0..tp-1). GGML backends. |
TENSORSHARP_TP_NODE_ID / TENSORSHARP_TP_PEERS | Multi-node distributed TP: 0-based node ID and the shared host:port peer list (= --tp-node-id / --tp-peers). |
TENSORSHARP_TP_CONNECT_TIMEOUT_SECONDS / _RECV_TIMEOUT_SECONDS | Peer connect retry window (120 s) and per-receive timeout (300 s) for the distributed TP mesh. |
TENSORSHARP_TP_DISABLE_P2P / TENSORSHARP_TP_HOST_ALLREDUCE | 1 forces cross-GPU copies / the local AllReduce through host memory instead of CUDA P2P (diagnostics). |
TS_GGML_TP_DEVICE_AR_THRESHOLD / TS_GGML_TP_PARALLEL | GGML TP: element count above which AllReduce uses the device collective (default 262144), and 0 to drive the ranks sequentially (diagnostic). |
TS_GEMMA4_TP_FUSED_MOE | 0 falls back from Gemma 4's fused MoE trunk under TP to the whole-expert per-op path. |
GGML_CUDA_ALLREDUCE / GGML_CUDA_AR_BF16_THRESHOLD | ggml collective selection (nccl / internal / none) and the payload size above which F32 collectives convert to BF16 (raised to 1 MB by TensorSharp). |
TS_KV_CACHE_REDIS_URL / TS_KV_CACHE_REDIS_TTL_MINUTES | Redis-backed shared KV cache tier and its entry TTL (default 1440 min; 0 = no TTL). |
TS_RESPONSES_STORE_REDIS_URL | Redis-backed OpenAI Responses API store (replaces the in-memory store). |
TS_GGML_VULKAN_DEVICE | Vulkan device index for ggml_vulkan (= --gpu-device). |
TS_FUSED_QKNORM_ROPE | 0 disables the fused QK-Norm + RoPE CUDA kernel in Qwen 3.5/3.6 text prefill on the direct cuda backend (default on). |
TS_JSON_FORCE_OPEN | 0 disables the structured-output first-token { constraint (on by default for json_object / json_schema). |
TS_MLX_* | MLX backend tuning: pipelined decode, mlock GGUF, fused KV write, batched MoE decode, memory caps. |
TENSORSHARP_MLX_LIBRARY / _LIBRARY_DIR | Override the search path for libmlxc. |
TENSORSHARP_GGML_NO_UPDATE / _GGML_GIT_REF | Skip / pin the ggml source clone on native builds. |
TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN | Force the GGML Vulkan backend on/off at native build time (ON/OFF = build-script --vulkan/--no-vulkan). Auto-enabled when a Vulkan runtime is detected; an explicit choice sticks across rebuilds. |
HTTP endpoints
| Method & path | Style | Purpose |
|---|---|---|
GET / | Utility | Plain liveness response. |
GET /index.html | Web UI | Browser chat application. |
POST /api/generate | Ollama | Single-prompt completion (stream or not). |
POST /api/chat/ollama | Ollama | Multi-turn chat with optional think / tools / images. |
GET /api/tags | Ollama | List the hosted model. |
POST /api/show | Ollama | Model info. |
POST /v1/chat/completions | OpenAI | Chat Completions (stream, tools, response_format). |
GET /v1/models | OpenAI | List models. |
POST /api/chat | Web UI | SSE chat stream with session + KV-reuse fields. |
POST /api/sessions · DELETE /api/sessions/{id} | Web UI | Create / dispose a per-tab session. |
POST /api/upload | Web UI | Upload an image / audio / video / PDF / text file. |
POST /api/image-edit | Web UI | Qwen-Image-Edit one-shot edit (multipart or JSON). |
POST /api/image-edit/stream | Web UI | SSE image edit with live denoising previews. |
GET /api/models | Web UI | Hosted model, supported backends, defaults. |
POST /api/models/load | Web UI | Re-load the startup model (optionally on a different backend). |
GET /api/version · GET /api/queue/status | Utility | Server version / legacy queue snapshot. |
Sampling parameters
Ollama (options) | OpenAI (top-level) | Default | Meaning |
|---|---|---|---|
num_predict | max_tokens | --max-tokens / 20000 | Maximum tokens to generate; the server default applies to every endpoint when omitted. OpenAI's max_completion_tokens is accepted too. |
temperature | temperature | 0.8 | Sampling temperature (0 = greedy). |
top_k | — | 40 | Top-K filtering (0 = disabled). |
top_p | top_p | 0.9 | Nucleus sampling threshold (1.0 = disabled). |
min_p | — | 0 | Minimum probability filtering. |
repeat_penalty | — | 1.1 | Repetition penalty (1.0 = off). |
presence_penalty / frequency_penalty | presence_penalty / frequency_penalty | 0 | Presence / frequency penalties. |
seed | seed | -1 | Random seed (-1 = random). |
stop | stop | null | Stop sequences. |
| — | response_format | null | text, json_object, or json_schema. |
C# public API
| Member | Signature / values | Notes |
|---|---|---|
ModelBase.Create | static ModelBase Create(string ggufPath, BackendType backend) | Auto-detects architecture from GGUF metadata. |
ModelBase.Forward | float[] Forward(int[] tokens) | Returns next-token logits (length = vocab size). |
ModelBase.Sample | int Sample(float[] logits, SamplingConfig config, IList<int> generated = null) | Applies penalties + sampling. |
ModelBase.SampleGreedy | int SampleGreedy(float[] logits) | Deterministic argmax. |
ModelBase.Config / .Tokenizer | ModelConfig / ITokenizer | Config.VocabSize, context length, etc. |
BackendType | Cpu, GgmlCpu, GgmlMetal, GgmlCuda, Cuda, Mlx | Backend selector enum. |
ITokenizer.Encode | Encode(string text, bool addSpecial) | Text → token ids. |
ITokenizer.Decode | string Decode(List<int> ids) | Token ids → text. |
ITokenizer.IsEos / .EosTokenIds | bool IsEos(int id) / int[] EosTokenIds | End-of-sequence detection. |
SamplingConfig | Temperature, TopK, TopP, MinP, penalties, Seed, StopSequences, MaxTokens | See C# Library. |
IBatchedPagedModel.ForwardBatch | batched/paged forward | Implemented by most architectures for continuous batching. |
REPL commands
| Command | Description |
|---|---|
/help, /? | Show all interactive commands. |
/exit, /quit | Leave the session. |
/reset, /new | Clear conversation history and KV cache. |
/history · /save <file> | Print / append the transcript. |
/system <text> | Set the system prompt (resets KV cache). |
/think on|off · /multiline on|off | Toggle reasoning mode / multi-line input. |
/info, /status | Show model, backend, architecture, context/vocab, projector, depth. |
/model <path> · /backend <name> · /mmproj <path> | Hot-swap model, backend, or projector. |
/sampling, /show | Print current sampling configuration. |
/max · /temp · /topk · /topp · /minp | Set reply length / temperature / top-k / top-p / min-p. |
/repeat · /presence · /frequency · /seed | Set penalties and seed. |
/stop <text> · /clearstop | Add / clear stop sequences. |
/image · /audio · /video · /text <path> · /clearattach | Attach media / text for the next turn; drop pending attachments. |