Advanced Features

The systems that make TensorSharp fast and scalable: continuous batching with a paged KV cache, speculative decoding, text diffusion, and the kernel- and memory-level optimizations underneath.

Continuous batching & paged KV cache

The server's InferenceEngine is a vLLM-style continuous-batching engine, on by default. Instead of running one request at a time, it interleaves many at the granularity of a single decode step.

• Paged KV pool — KV cache is partitioned into fixed-size blocks drawn from a shared pool, so memory is allocated per block rather than per worst-case sequence length.
• Block-hash prefix sharing — each full block is content-hashed; identical prefixes (system prompts, shared context) are shared across concurrent and sequential requests instead of recomputed.
• Iteration-level scheduler — admits and preempts sequences mid-batch and packs them into one forward pass on models that implement IBatchedPagedModel.
• Optional SSD tier — cold blocks can spill to an SSD tier for very large KV working sets.

Models that have not implemented the batched path still run on the engine's isolated per-sequence KV-swap fallback. Tune it with the TS_SCHED_* variables, or disable it entirely with --no-continuous-batching.

Native paged attention

The native kernel TSGgml_PagedAttentionForward (and a WithSinks variant for GPT OSS) gathers K/V from the paged buffer in C++, builds a small GGML graph per sequence, and dispatches ggml_flash_attn_ext — the same fused Metal/CUDA flash-attention kernel the single-sequence path uses. On a long-context Ministral-3-14B workload (4×~800 tokens) it runs ~21% faster than the legacy per-sequence GGML path.

Batched forward passes (Mistral 3, Gemma 4, GPT OSS, Qwen 3.5/3.6 with a GatedDeltaNet recurrent-state pool, and Nemotron-H with a Mamba2 recurrent-state pool) pack N sequences into one ForwardBatch call with one batched linear-projection matmul per layer and paged K/V scatter. Gemma 4 reaches ~1.5–1.6× legacy throughput; Nemotron-H Mamba2 batched reaches ~3.95× at batch=3 on an Apple M4 Pro.

MTP / NextN speculative decoding

Some architectures ship a multi-token-prediction (MTP / NextN) draft head that lets the server run lossless speculative decoding for solo (non-concurrent) sequences. The draft proposes several future tokens cheaply, the trunk verifies all of them in one batched forward, and accepted tokens are committed in a single step.

It is off by default. Enable it on the server with --mtp-spec (env TS_MTP_SPEC=1):

# Qwen 3.6 — the NextN block is embedded in the trunk GGUF, no extra file needed
# (use a GGUF from unsloth/Qwen3.6-35B-A3B-MTP-GGUF; base-repo exports strip the NextN block)
dotnet run --project TensorSharp.Server -c Release --no-build -- --model Qwen3.6-35B-A3B-UD-IQ2_XXS.gguf --backend ggml_cuda \
    --mtp-spec --mtp-draft 8 --mtp-pmin 0.75

# Gemma 4 — load the separate gemma4-assistant draft GGUF that matches the target
dotnet run --project TensorSharp.Server -c Release --no-build -- --model gemma-4-12B-it-qat-UD-Q4_K_XL.gguf --backend ggml_cuda \
    --mtp-spec --mtp-draft-model mtp-gemma-4-12B-it.gguf

Two draft-head shapes

• Qwen 3.6 (embedded NextN) — the GGUF carries one extra decoder block plus the NextN projection/norm tensors. No separate file; --mtp-draft-model is ignored. Note that only GGUFs that retain the NextN tensors carry it — the unsloth/Qwen3.6-35B-A3B-MTP-GGUF repo keeps them, while base-repo Qwen3.6 exports strip the block and silently fall back to standard decode. The GatedDeltaNet trunk state is snapshotted so a partially-rejected verify batch can roll back.
• Gemma 4 (separate gemma4-assistant GGUF) — an EAGLE-style recurrent drafter loaded with --mtp-draft-model. It holds no K/V of its own: every draft layer queries the target model's existing per-layer KV cache. The draft's hidden size must match the target (pair the 12B target with its 12B draft). A mismatched or incomplete draft fails fast at startup with a remediation hint.

Where it's profitable

Tuning: --mtp-draft (default 8) bounds tokens drafted per step; --mtp-pmin is the minimum draft confidence to keep a token — its default depends on the drafter kind (0.75 for a per-token draft head, 0.35 for a block drafter, where the gate is the cumulative prefix probability and so the same number is far stricter). Gemma 4 A/B switches are the TS_GMTP_* env vars.

DSpark block speculative decoding (DeepSeek V4)

DeepSeek V4 ships DSpark ("Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation") as a support module in the checkpoint: three DSV4 blocks that read the trunk's hidden states and propose a whole block of tokens per step instead of one, a Markov head that conditions each block position on the token before it, and a confidence head that predicts each position's acceptance probability. The trunk then verifies the block in one batched forward and keeps only the prefix its own sampler would have produced.

The drafter is a separate GGUF loaded with --draft-model — every GGUF conversion of the trunk drops the mtp.* tensors. Pre-built drafters are listed in the repository's MODEL_DOWNLOADS.md, or you can convert one from the upstream safetensors checkpoint with eng/dsv4-dspark-to-gguf.py (only the three shards holding mtp.* are downloaded).

# CLI — every single-sequence path (--input, --multi-turn-jsonl, --interactive) uses it
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model DeepSeek-V4-Flash-...-00001-of-00005.gguf \
    --backend ggml_cuda --draft-model DSpark-drafter-Q2K-Q8-0731.gguf \
    --input prompt.txt --max-tokens 200 --temperature 0

# Server — same drafter, plus --mtp-spec
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model DeepSeek-V4-Flash-...-00001-of-00005.gguf \
    --backend ggml_cuda --tp 4 --mtp-spec --draft-model DSpark-drafter-Q2K-Q8-0731.gguf

It runs on both GPU engines — --backend cuda (direct CUDA) and --backend ggml_cuda (native ggml); ggml_vulkan and cpu have no speculative path for this architecture and log a warning if a drafter is configured. On ggml the drafter is three extra graph layers whose key ring the trunk graph commits itself, so speculation costs no host round-trips. On the CLI it needs a pure-argmax sampler (any temperature, top-k/p, or repetition penalty turns it off); on the server every verify row is drawn with the request's own sampler, so it composes with any sampling settings. Speculation serves solo sequences only — as soon as a second request is in flight, DSV4's per-sequence slots serve the batch at normal decode speed.

Measured (4×A40, DeepSeek-V4-Flash-0731 UD-Q8_K_XL, greedy)

Multi-turn chat benefits most — a turn that continues an established context is exactly where the drafter is confident. On a 5-turn interactive session the same box measured 1.50× to 2.02× per turn (66–87% acceptance), with prefill at parity. Greedy output was byte-identical to the non-speculative baseline on both the 200-token and the 15K-context runs.

Why 1.3× and not more: the trunk is 6-of-256 sparse, so each extra token in the verify batch pulls its own set of routed experts through VRAM — a verify row costs roughly a quarter of a full decode step no matter how cheap the draft was. --spec-draft-conf-min (default 0.35, the minimum cumulative acceptance probability for a drafted position) is the knob that keeps that trade positive; --spec-draft-n-max caps tokens per block.

DiffusionGemma text diffusion

DiffusionGemma is fundamentally different from autoregressive models: it does not call Forward() one token at a time. Instead it uses block-wise EntropyBound denoising over fixed-length canvases on a Gemma-4-derived MoE backbone — the whole answer is refined iteratively rather than written left to right.

• CLI — --diffusion-steps (denoising steps per block), --diffusion-seed, and --diffusion-blocks.
• Web UI — streams whole-message replace events so you watch the answer denoise live, and batches concurrent diffusion requests at block boundaries via DiffusionBatchScheduler.
• Optimizations — on GPU backends the prompt side of [prompt | canvas] is prefetched once per block and reused across steps; GGML backends use a fused whole-model diffusion decode plus a fused lm-head tail.

Performance optimizations

A cross-architecture summary; each per-model card in docs/models/ walks through the same kernels with the exact GGML graph dispatched.

• Single-graph GPU decode (Gemma 4 dense, including E4B) — all transformer layers run in one GGML graph dispatch, cutting CPU↔GPU round-trips from hundreds per token to one (~2.6× over per-op dispatch).
• Whole-model fused decode graphs (Gemma 4 dense + MoE, Qwen 3.5/3.6, GPT OSS) — an entire decode token, including the MoE router and experts, the final norm, and the LM head, is submitted as one GGML graph. On CUDA/Vulkan the graph is built once with stable tensor addresses and replayed, which is what lets ggml-cuda capture it as a CUDA graph. GPT OSS decode goes from 24 → 154 tok/s on an A40 and stays flat in context length (133 tok/s at 16K) where the per-layer path collapsed to 2.3. Per-model opt-outs: TS_GPTOSS_MODEL_DECODE=0, TS_GEMMA4_FD_PERSIST=0, TS_QWEN35_FD_PERSIST=0.
• DeepSeek V4 whole-model executors — deepseek4 bypasses the generic per-op forward entirely. Each executor loads the split GGUF itself, layer-splits the weights across every visible GPU, keeps all DSV4 KV state on-device (raw SWA ring, CSA/HCA compressed-K caches, lightning-indexer cache), and runs each prefill/decode micro-batch as a single graph with a shape-signature cache, so steady-state decode replays a captured CUDA graph. Decode attention gathers a compact [ring | top-512] K through a fused index-gather instead of scanning the whole context.
• Device-side Nemotron-H decode attention — on GGML backends the attention layers decode through the flash-attention kernel against the resident KV cache (TS_NEMOTRON_FLASH_DECODE=0 restores the host path), so decode no longer degrades with context length.
• Fused prefill / verify with E4B semantics — the native whole-model path carries per-layer embeddings (PLE) and shared-KV donor mapping through E4B prefill/verify instead of falling back to hundreds of per-op submissions.
• Automatic scheduler N=1 fast path — when only one sequence is scheduled, the server automatically uses Gemma 4's fused per-sequence forward rather than paying the fully batched path's fixed overhead.
• Chunked prefill (Gemma 4) — long prompts are split into bounded chunks to avoid O(n²) attention score tensors for sliding-window layers. A solo (uncontended) request runs through the fused whole-graph path with one larger cap, TS_SCHED_SOLO_PREFILL_CHUNK (default 8192); under contention the chunk size drops to TS_SCHED_PREFILL_CHUNK (default 1024).
• Fused Qwen 3.5/3.6 attention & FFN — single-graph fused attention-layer decode, fused prefill attention, fused out-proj + FFN, and fused vision encoder blocks (~15 ops → 2).
• Fused QK-Norm + RoPE (Qwen 3.5/3.6, direct CUDA) — a single CUDA kernel applies QK-Norm and NeoX RoPE in the text-only prefill path on the cuda backend. Default on; disable with TS_FUSED_QKNORM_ROPE=0.
• Native quantized compute — Q4_K_M, Q6_K, Q8_0, IQ2_XXS, MXFP4 used directly in matmul without expanding to FP32; a batched AddmmQuantBatch handles multiple sub-weight matmuls in one dispatch.
• Batched GPU MoE — all selected experts (plus the optional shared expert and residual add) collapse into a single GGML graph dispatch per MoE layer.
• KV-cache prefix reuse — multi-turn conversations reuse the longest matching token prefix; sliding-window models back off truncation by the window size.
• Kernel warmup — both CLI and server run a tiny forward pass at startup to pre-compile GPU kernels and warm the pool, avoiding cold-start latency.

Memory optimizations

• Zero-copy file-mapped weights — the GGUF is memory-mapped and quantized tensors bind directly into native ops, removing a per-tensor copy that roughly doubled the resident set. Example: Qwen3.5-35B-A3B-IQ2_XXS (~10 GB GGUF) runs at ~7 GB peak under Metal instead of ~17 GB.
• Best-fit memory pool with bounded retention (blocks capped at 64 MB, pool at 32 blocks) keeps the working set tight across long runs.
• Paged KV block pool with optional SSD spillover — RAM-capped, LRU-evicted, with content-hash prefix reuse across sessions.
• KV block codecs — optional in-place compression with TurboQuantKvCodec (Q2 / Q4 / Q8) via --paged-kv-quant-bits, trading a small accuracy cost for half/quarter the per-block footprint.