Features

A complete catalog of what TensorSharp does today. Each item links to the page where you can use it.

Highlights

Models & modalities

• Multi-architecture support — DeepSeek V4 Flash, Gemma 4, Gemma 3, DiffusionGemma, Qwen 3, Qwen 3.5/3.6-family, GPT OSS, Nemotron-H, Mistral 3, Qwen-Image-Edit. → Supported models
• DeepSeek V4 Flash (284B MoE) — a compressed-sparse-attention, 1M-context architecture with three dedicated whole-model executors: a direct-CUDA engine (--backend cuda), the native ggml executor (ggml_cuda / ggml_vulkan), and a 100% pure-C# CPU executor (--backend cpu, no native dependencies). Weights layer-split automatically across every visible GPU, and the server hosts it with native per-sequence slots and continuous batching. → DeepSeek V4
• Multimodal inference — image, video, and audio inputs for Gemma 4; images for Gemma 3, Qwen 3.5-family, Mistral 3, and Nemotron-H Omni. → Multimodal
• PDF document input — born-digital PDFs have their complete text layer extracted and inlined into the prompt; scanned PDFs fall back to page images for vision-capable models. Available as a Web UI upload and via the CLI's one-shot --pdf flag; cap the pages read with TS_PDF_MAX_PAGES (default: all). → Web UI
• Mixture of Experts (MoE) — Gemma 4 MoE (e.g. 26B-A4B), GPT OSS MoE (gpt-oss-20b), Qwen 3.5/3.6 MoE (35B-A3B), and Nemotron-H MoE FFN layers, with a fused batched GPU MoE dispatch.
• Hybrid SSM-Transformer — Nemotron-H mixes Mamba2 SSM layers, attention layers, and MoE FFN in one model.
• Hybrid Attention-Recurrent — Qwen 3.5/3.6-family mix full-attention layers with GatedDeltaNet recurrent layers.
• Text-diffusion generation — DiffusionGemma uses an iterative EntropyBound denoising sampler instead of autoregressive decode. → DiffusionGemma
• Image editing (Qwen-Image-Edit) — a prompt + input image produces an edited image via a 60-block MMDiT diffusion transformer, a Qwen-Image VAE, and a Qwen2.5-VL-7B text encoder (FlowMatch-Euler true-CFG, CUDA-graph-captured DiT forward); an optional Lightning distillation LoRA (--qwen-image-lora) cuts the denoise loop to a few steps at CFG 1.0. → Image editing

Generation & control

• Thinking / reasoning mode — structured chain-of-thought with <think> / <|channel> tags (Qwen 3, Qwen 3.5/3.6, Gemma 4, GPT OSS, Nemotron-H). → Thinking mode
• Tool calling / function calling — architecture-agnostic output parsing turns raw model output into structured tool_calls, whether the model emits JSON (Qwen 3, Nemotron-H), XML inside a <tool_call> block (Qwen 3.5/3.6), Harmony commentary (GPT OSS), or DSML markup (DeepSeek V4). → Tool calling
• Configurable sampling — temperature, top-k, top-p, min-p, repetition / presence / frequency penalties, seed, and stop sequences. → Sampling
• Structured outputs — OpenAI response_format with text, json_object, and validated json_schema. → Structured outputs
• Chat templates — auto-loaded from GGUF metadata (Jinja2), with hardcoded fallbacks per architecture.
• Streaming — token-by-token output via SSE (web) or stdout (console), with abort/stop support for in-flight generations.

Performance & scale

• Trades wins with llama.cpp head-to-head — on identical GGUF files and the same GPU, this pure-.NET engine matches or beats the hand-tuned C++ llama.cpp where it counts. In the current comparison run (short/long/multi-turn text on both engines' GGML CUDA and Vulkan builds, reproducible via benchmarks/engine_comparison): Gemma 4 E4B and the 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner (multi-turn prompts up to 1.49×), Gemma 4 12B decodes 1.21× faster on Vulkan (up to 1.32× on long context), and CUDA decode holds parity or better on three of four models (up to 1.07× geomean on Qwen 3.6 27B). An earlier CUDA image_edit run also measured TensorSharp's Qwen-Image-Edit finishing a warm edit in 40.44 s vs 48.16 s for stable-diffusion.cpp (~1.19× faster). → Head-to-head benchmarks
• GPU-accelerated — GGML Metal (macOS), GGML CUDA (Windows/Linux + NVIDIA), GGML Vulkan (Windows/Linux + AMD/Intel/NVIDIA), a direct CUDA/cuBLAS backend, and an MLX backend for Apple Silicon — all with CPU fallbacks. → Backends
• Continuous batching & paged KV cache — block-paged KV pool with block-hash prefix sharing, an iteration-level scheduler that admits/preempts sequences mid-batch, optional SSD-backed tier, and a native fused paged-attention kernel. → Deep dive
• Batched / parallel inference — N sequences packed into a single forward pass with paged K/V scatter (Mistral 3, Gemma 4, GPT OSS, Qwen 3 / 3.5 / 3.6, Nemotron-H).
• Tensor parallelism & distributed inference — split one model across N GPUs with --tp N on the CLI or the server (Megatron-LM column/row-parallel, replicated norms/embeddings/LM head), on the direct cuda backend and on GGML CUDA / Vulkan, and extend the group across machines with a peer-to-peer TCP mesh (--tp-node-id / --tp-peers). Hierarchical AllReduce keeps only 1/tp_local of each collective on the network; MoE expert slicing and expert parallelism, GatedDeltaNet per-rank V-head ownership, and Mamba2 replication cover the heterogeneous layers. Fused per-rank block graphs put --tp 2 decode above a single GPU on Gemma 4 (51.7 vs 37.3 tok/s) and run models too large for one card. → Multi-GPU & Multi-Node
• MTP / NextN speculative decoding — multi-token-prediction draft heads accelerate solo decode; lossless because the request's own sampler drives both draft and verify. → Speculative decoding
• DSpark block speculative decoding — DeepSeek V4's drafter proposes a whole block of tokens per step (a Markov head conditions each block position on the one before it, a confidence head gates how far to draft) and the trunk verifies the block in one batched forward. Loaded as a separate GGUF with --draft-model; measured 1.3–1.4× decode on 4×A40, up to 2.0× on multi-turn chat, with greedy output byte-identical to the baseline. → DSpark
• Whole-model fused decode graphs — Gemma 4, Qwen 3.5/3.6, and GPT OSS submit an entire decode token as one GGML graph instead of one dispatch per layer, so the GPU never waits on the host between layers. GPT OSS decode: 24 → 154 tok/s on an A40, and flat in context length. → Performance optimizations
• Native quantized compute — quantized weights are used directly in matmul without expanding to FP32, saving memory and bandwidth.
• Optimized pure C# CPU backend — managed GEMM fast paths plus fused SIMD kernels for RMSNorm, RoPE, softmax, and fused activations.
• KV cache codecs — pluggable IKvBlockCodec with a built-in TurboQuant (Q2 / Q4 / Q8) compressed codec for paged blocks.

Interfaces & integration

• Ollama & OpenAI API compatibility — drop-in replacement endpoints for existing tooling. → HTTP API
• Browser chat UI — multi-turn chat, file uploads up to 500 MB (images, video, audio, PDF, text/code), thinking toggle, tool calling, message editing, and live streaming. → Web UI
• Interactive REPL — a turn-by-turn console chat with slash commands, hot-swappable model/backend/projector, and live sampling tuning. → REPL
• Batch processing — JSONL input in the console application, plus a built-in prefill/decode benchmark.
• Source-project embedding — reference only the layers you need and embed inference in your own .NET app. Current Runtime/Models/Backends packages are not yet on NuGet.org. → C# Library
• Per-turn observability — structured bounded input summaries, raw output logs, and KV-cache hit ratios surfaced through every API (prompt_cache_hit_*, cached_tokens, kvReused*). Uploaded document bodies are omitted from logs while their metadata and user instruction remain visible.