TensorSharp
A native .NET LLM inference engine for GGUF models — with a command-line tool, a browser chat server, and Ollama- & OpenAI-compatible APIs for programmatic access.
Everything runs on your own hardware: your laptop, workstation, or server. No data leaves the machine, there are no per-token fees, and the same engine powers a quick command-line test, a shared internal chatbot, and a production REST endpoint. This wiki is the complete guide — pick a starting point below or use / to search.
The TensorSharp book · Now available
From Tensors to Tokens
Want to understand the engine, not just run it? Follow the full path from tensor layouts to streamed multimodal tokens as you build, verify, and optimize a modern C#/.NET inference engine with TensorSharp and Gemma 4 E4B.
Explore the wiki
Getting Started
Prerequisites, build, download a model, and stream your first reply.
Command Line
Run prompts, images, audio, batches, and benchmarks from the CLI.
Server & Web UI
Host a browser chatbot and HTTP endpoints on localhost.
HTTP API
Call it from curl, Python, or any Ollama/OpenAI client.
C# Library
Embed the engine directly in your .NET application.
API Reference
Searchable tables of flags, env vars, endpoints, and types.
Models
Supported architectures, downloads, multimodal, and reasoning.
Multi-GPU & Multi-Node
Tensor parallelism across GPUs — CUDA and GGML — and across machines.
Glossary & FAQ
New to LLMs? Plain-language definitions and common questions.
Quick start in ~30 seconds
After installing the .NET 10 SDK for your platform, git, and curl, this path runs the benchmark-verified Gemma 4 E4B Q8_0 model on a native GGML backend. Copying the commands takes about 30 seconds; the 7.48 GiB model download and the first restore/build take longer and depend on your connection and machine. This block is for Linux + NVIDIA — see other backends below. On Windows PowerShell, use curl.exe.
-
Clone
The first
dotnet runbelow restores the projects and compiles the native GGML CUDA backend; that first build can take several minutes.git clone https://github.com/zhongkaifu/TensorSharp.git cd TensorSharp -
Download the model
The recommended artifact is
gemma-4-E4B-it-Q8_0.gguf(7.48 GiB); the lower-memorygemma-4-E4B-it-Q4_K_M.gguflives in the same repository.curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf -
Run it
The environment assignment enables the CUDA native build. E4B also supports thinking (
--think) and tool calling (--tools).echo "What is TensorSharp? Answer briefly." > prompt.txt TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda -
Prefer a UI + API?
Start the server with the same model. The health check is at
/; the browser chat is at/index.html.TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet run --project TensorSharp.Server -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 128 # open http://localhost:5000/index.html
Other backends and multimodal
On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan; without a GPU, use ggml_cpu (native CPU kernels, no GPU toolchain). Text inference needs only the model GGUF. For image, video, or audio input, also download the matching mmproj-gemma-4-E4B-it-Q8_0.gguf and pass it with --mmproj. Windows PowerShell and full platform syntax are in the E4B fast-lane notes.
Why TensorSharp?
Private by default
Inference happens on your hardware. Prompts, documents, and images never leave the machine.
No per-token bill
Run as much as your hardware allows — predictable cost, no metered API.
Drop-in compatible
Speaks the Ollama and OpenAI wire formats, so existing tools and SDKs just work.
Runs anywhere
NVIDIA (CUDA), AMD / Intel / NVIDIA (Vulkan), Apple Silicon (Metal/MLX), or pure CPU — with automatic fallbacks.
Modern model support
Gemma, Qwen, GPT-OSS, Nemotron-H, Mistral, plus vision, audio, PDF documents, reasoning & tools.
Built in .NET
A native C# engine you can embed in your apps, not just a black-box binary.
Scales past one GPU
Tensor parallelism splits a model across several GPUs with --tp N — on the direct CUDA backend and on GGML CUDA / Vulkan — and across machines over a peer-to-peer TCP mesh when one host is not enough.
Benchmarked vs llama.cpp
On identical GGUF files and the same GPU it trades wins with the C++ engine — in the current CUDA + Vulkan comparison run (reproducible via benchmarks/engine_comparison): Gemma 4 E4B and the 2-bit Qwen 3.6 35B-A3B MoE prefill 1.28× faster on CUDA with first tokens 1.27× sooner, multi-turn prompts prefill faster on every model (up to 1.49×), and Gemma 4 12B decodes 1.21× faster on Vulkan.
Who is this for?
TensorSharp serves a wide range of visitors. Here is the fastest path for each.
Beginners & students
Start with the guided book or Glossary & FAQ, then try Getting Started.
Developers
Jump to the HTTP API, C# Library, and API Reference.
Senior / principal engineers
Read Advanced Features — paged KV, continuous batching, speculative decoding.
Managers, CTOs & CEOs
See the business value and capability matrix.
Sales & marketing
Use the feature catalog and benchmarks for positioning.
Researchers & professors
Explore model architectures and the head-to-head benchmarks.