Benchmarks & Testing
How TensorSharp measures up against other inference engines on the same models and hardware, plus the test harness that guards correctness.
Head-to-head: TensorSharp vs llama.cpp
A pure-.NET engine going toe-to-toe with the hand-tuned C++ llama.cpp — same GGUF files, same NVIDIA RTX 3080 Laptop GPU (16 GB), one uniform OpenAI /v1/chat/completions surface, and both engines measured on their GGML CUDA and GGML Vulkan builds across short-prompt, long-context, and multi-turn chat scenarios. The figures below are the current checked-in run of the benchmarks/engine_comparison harness (docs/engine_comparison_report.md), reproducible on your own hardware.
CUDA prefill & TTFT
Gemma 4 E4B prefills 1.28× faster on geomean (up to 1.39× on short prompts); the 2-bit Qwen 3.6 35B-A3B MoE matches it at 1.28× prefill / 1.27× TTFT.
Multi-turn prefill sweep
On CUDA, follow-up turns prefill faster on every model — 1.27× (E4B), 1.39× (12B), 1.49× (35B-A3B), 1.11× (27B) — with first tokens up to 1.47× sooner.
CUDA decode: parity or better
E4B 1.02×, 12B 1.04×, 27B 1.07× geomean — long-context decode up to 1.12× — and the 35B-A3B MoE within 2% (0.98×).
Vulkan dense decode 1.21×
Gemma 4 12B decodes 1.21× faster on the vendor-neutral Vulkan path (up to 1.32× on long context), with prefill/TTFT at or above parity.
Geomean speedup of TensorSharp over llama.cpp on the same backend (single-stream, greedy, MTP off). > 1.0× means TensorSharp is faster (decode / prefill) or lower-latency (TTFT):
| Model | Backend | decode | prefill | TTFT |
|---|---|---|---|---|
| Gemma 4 E4B it (Q8_0, dense multimodal) | CUDA | 1.02× | 1.28× | 1.27× |
| Gemma 4 E4B it (Q8_0, dense multimodal) | Vulkan | 1.00× | 1.05× | 1.03× |
| Gemma 4 12B it (UD-Q4_K_XL, dense) | CUDA | 1.04× | 1.17× | 1.16× |
| Gemma 4 12B it (UD-Q4_K_XL, dense) | Vulkan | 1.21× | 1.04× | 1.03× |
| Qwen 3.6 35B-A3B (UD-IQ2_XXS, MoE) | CUDA | 0.98× | 1.28× | 1.27× |
| Qwen 3.6 35B-A3B (UD-IQ2_XXS, MoE) | Vulkan | 0.87× | 1.04× | 1.03× |
| Qwen 3.6 27B (UD-IQ2_XXS, dense) | CUDA | 1.07× | 0.96× | 0.95× |
| Qwen 3.6 27B (UD-IQ2_XXS, dense) | Vulkan | 1.02× | 0.85× | 0.84× |
The headline wins come from TensorSharp's architecture-specific fast paths: fused whole-model prefill/verify graphs with fused FFN / attention kernels, fused whole-model decode, and the vLLM-style paged KV cache with cross-request prefix sharing. Even at extreme quantization the engine holds its own: on 2-bit IQ2_XXS weights the dense Qwen 3.6 27B decodes 1.07× (CUDA) / 1.02× (Vulkan) faster and the 35B-A3B MoE out-prefills llama.cpp by 1.28× — a pure-.NET engine keeping pace with hand-tuned C++. The remaining sub-1.0× cells — chiefly Qwen 3.6 35B-A3B decode on Vulkan (0.87×), Qwen 3.6 27B dense prefill (0.96× CUDA / 0.85× Vulkan), and E4B long-prompt prefill on Vulkan (0.71×) — are active optimization targets rather than finished results.
Beyond the text scenarios — the harness also supports tool-calling, structured-output (JSON), image / audio / video, MTP on/off, and parallel-request-scaling scenarios against the same servers, plus an image_edit scenario against stable-diffusion.cpp; an earlier CUDA run of that scenario (Qwen-Image-Edit 2511 Q2_K DiT + Lightning 4-step LoRA, 544×1184, 4 steps, identical inputs and seed) measured TensorSharp completing a warm edit in 40.44 s vs 48.16 s — ~1.19× faster. Enable any of these cells with the harness flags documented in benchmarks/engine_comparison.
An honest comparison shows the gaps as well as the wins. These figures are one machine (RTX 3080 Laptop, single-stream); your hardware, model, and quantization will shift them. Run the built-in benchmark and the engine-comparison harness on your own box for the numbers that matter to you.
Testing
The build and test commands below require the .NET 10 SDK for your platform; verify the installation before running them.
Unit tests (xUnit)
InferenceWeb.Tests exercises in-process behavior that doesn't require a running server: managed quantized ops, direct CUDA and MLX backend kernels (when the hardware is available), paged-KV scheduling, batched-executor correctness, per-model batched-forward correctness against the legacy path, MTP / NextN speculative-decoding correctness, DiffusionGemma probes, codec round-trips, prompt rendering, and the server CLI options builder.
dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csprojServer integration tests
Integration tests in TensorSharp.Server/testdata/ cover all three API styles (Web UI SSE, Ollama, OpenAI), multi-turn conversations, thinking mode, tool calling, structured outputs, queue-status compatibility, concurrent requests, and abort support. Architecture-specific features are auto-detected and skipped when the active model doesn't support them.
# Start TensorSharp.Server, then run:
python3 TensorSharp.Server/testdata/test_multiturn.py
# or
bash TensorSharp.Server/testdata/test_multiturn.shInference matrix runner
TensorSharp.TestMatrix is the broader CLI-driven harness for long-running model × backend × feature × env-var coverage. It discovers GGUF files, filters unavailable backends and unsupported prompt types, runs baseline plus env-var sweep cells, writes one JSON result per cell, emits an aggregate Markdown report, and compares against per-host baselines.
dotnet build TensorSharp.TestMatrix/TensorSharp.TestMatrix.csproj -c Release
dotnet run --project TensorSharp.TestMatrix -c Release -- --dry-runBenchmark numbers depend heavily on hardware, model, quantization, and KV-cache dtype. Treat the figures above as a reproducible reference point on one machine, not a universal guarantee — run the built-in benchmark on your own hardware for the numbers that matter to you.