Benchmarks & Testing

How TensorSharp measures up against other inference engines on the same models and hardware — llama.cpp on text, stable-diffusion.cpp on video generation — plus the test harness that guards correctness.

Head-to-head: TensorSharp vs llama.cpp

TensorSharp’s .NET runtime with native GGML execution compared with llama.cpp — same GGUF files, same NVIDIA RTX 3080 Laptop GPU (16 GB), one uniform OpenAI /v1/chat/completions surface, and both engines measured on their GGML CUDA and GGML Vulkan builds across short-prompt, long-context, and multi-turn chat scenarios. The figures below are the current checked-in run of the benchmarks/engine_comparison harness (docs/engine_comparison_report.md), reproducible on your own hardware.

Geomean speedup of TensorSharp over llama.cpp on the same backend (single-stream, greedy, MTP off). > 1.0× means TensorSharp is faster (decode / prefill) or lower-latency (TTFT):

The headline wins come from TensorSharp's architecture-specific fast paths: fused whole-model prefill/verify graphs with fused FFN / attention kernels, fused whole-model decode, and the vLLM-style paged KV cache with cross-request prefix sharing. Even at extreme quantization the engine holds its own: on 2-bit IQ2_XXS weights the dense Qwen 3.6 27B decodes 1.07× (CUDA) / 1.02× (Vulkan) faster and the 35B-A3B MoE out-prefills llama.cpp by 1.28× — a result specific to these models and native GGML backends. The remaining sub-1.0× cells — chiefly Qwen 3.6 35B-A3B decode on Vulkan (0.87×), Qwen 3.6 27B dense prefill (0.96× CUDA / 0.85× Vulkan), and E4B long-prompt prefill on Vulkan (0.71×) — are active optimization targets rather than finished results.

Beyond the text scenarios — the harness also supports tool-calling, structured-output (JSON), image / audio / video, MTP on/off, and parallel-request-scaling scenarios against the same servers, plus an image_edit scenario against stable-diffusion.cpp; an earlier CUDA run of that scenario (Qwen-Image-Edit 2511 Q2_K DiT + Lightning 4-step LoRA, 544×1184, 4 steps, identical inputs and seed) measured TensorSharp completing a warm edit in 40.44 s vs 48.16 s — ~1.19× faster. Enable any of these cells with the harness flags documented in benchmarks/engine_comparison.

Apple Silicon: ggml_metal vs llama.cpp

ggml_metal is the production backend on macOS and iOS, so it is measured against llama.cpp the same way. On an Apple M5 Pro (20-core GPU, 48 GB, macOS 26.6), both engines run the same GGUF back to back under caffeinate on an idle machine — TensorSharp.Cli --backend ggml_metal --benchmark against llama-bench, three reps each.

Ratios are TensorSharp relative to llama.cpp; > 1.0× means TensorSharp is faster. The run found four graph-construction gaps on TensorSharp's side — a residual add written in the order that declines ggml's norm+add fusion, a fused QKV split with copies instead of strided views, and a saturated sliding window read rotated and rebuilt every token — worth up to +12.6% on a single cell (gemma-4-E4B tg128@4096) with no cell regressing. It also records two changes that did not pay, so they are not retried. Full method, per-cell numbers, and the correctness bug found on the way: docs/perf/metal-vs-llama-cpp.md.

Muse-Glimmer 30B, with and without DFlash

A separate, hand-run comparison on different hardware: one RTX PRO 6000 Blackwell (96 GB), Muse-Glimmer-30B-Q8_0, greedy on both sides, 128 generated tokens, two reps with the engines alternating, and — importantly — the same token sequence on both sides (TensorSharp's rendered chat prompt is handed to llama-cli -no-cnv, and llama-tokenize confirms the counts match). llama.cpp is master 8e7f22b, within a day of TensorSharp's vendored ggml.

The pattern: TensorSharp's fused whole-model graph wins prefill below ~2K by 1.16-1.27×, the engines cross somewhere between 2K and 16K, and llama.cpp keeps a 6-8% prefill edge above that; decode is a tie at short context and drifts to 0.86× at 128K. With DFlash speculative decoding TensorSharp is ahead at short and mid prompts and behind from 16K up — but that gap is mostly a runtime cost governor that parks the drafter after a mis-measured probe: on the reps where it stays armed, TensorSharp reaches 94% of llama.cpp at 16K and 96% at 64K. The absolute DFlash figures also depend heavily on the corpus (the 501-token prompt hits 100% draft acceptance on both engines). Both engines run the full 128K context on one card. One caveat on the two long rows: they were taken before a line-ending fix landed in the harness, so on those points TensorSharp prefilled the CRLF form of the same document (1.2% more tokens); throughput is a rate so the effect is small, but the generated continuations are not strictly comparable there. Method, per-rep spread, VRAM and the governor analysis are in the Muse-Glimmer architecture card (docs/models/muse-glimmer.md).

GLM 5.x: the other end of the size range

A second hand-run comparison, this one on a model no single card holds: 3× RTX PRO 6000 Blackwell (97 GiB each, PCIe, no NVLink), GLM-5.2-UD-IQ2_XXS (744B-A40B MoE, MLA + DeepSeek Sparse Attention, 218 GiB of weights layer-split across the three cards), both engines measured back to back in one session — llama-bench for llama.cpp, and the parity harness's --bench, which reports the best of two repetitions the same way. Run-to-run spread is about 4%.

The crossover sits around a thousand prompt tokens, and the micro-batch is why: with 256 experts at top-8, a 512-token chunk routes only ~16 rows to each expert, so most of every expert-GEMM tile is padding and a bigger chunk buys more than anything else on the graph. Below that a whole prefill is one small graph, and the fixed cost per call — the managed hop, the input uploads, the 154,880-wide logits copy back — is a visible fraction of it, which is where llama.cpp's few percent come from. Decode is memory-bound and lands a few percent ahead either way. Output is checked as well as timed: on the same backend TensorSharp reproduces llama.cpp token-for-token on 6/6 recorded prompts on ggml_cuda, 3/3 on ggml_cpu, and 1/1 on the 100% managed cpu backend.

Two things do not speed this model up on this host. --tp 3 takes pp2048 / tg64 from 915.9 / 43.9 to 505.6 / 17.6 — each of the 78 layers needs two all-reduces of a [6144, n_tokens] hidden state, and PCIe with no NVLink cannot pay for that. --n-cpu-moe 30 takes it to 94.7 / 16.4. Both are there so a checkpoint that would not otherwise fit runs at all: offload raises the context the loader can size from 342,272 to 646,400 tokens. See GLM 5.x.

GLM-5.3 on eight A40s: a decode tie and a 2.9× faster load

GLM-5.3 (not Flash) is the same glm-dsa block shape as GLM-5.2 — 79 blocks (78 trunk + one NextN), 256 routed experts at top-8 plus one shared expert, MLA with the lightning indexer, rope base 8e6 — so it loads on the path above with no new code and no new flag, and because llama.cpp registers glm-dsa there is a legitimate reference column for it on a mainline build (GLM-5.3-Flash's glm5next needed an unmerged llama.cpp build, 2e0e57f / PR #27754, to compare at all). Measured on eight A40 46 GB (no NVLink, CUDA 12.8), GLM-5.3 UD-Q2_K_XL, a 10,531-token prompt and 300 decode tokens, median of three repeats with a fresh prompt body each time, whole-layer placement on both engines:

Decode is a tie inside a percent, and TensorSharp loads the 236.4 GiB checkpoint 2.9× faster. The honest gap is time to first token — 41.9 s against 29.0 s, about 1.4× slower — and it is the clearest actionable finding of that comparison. The llama.cpp prefill cell is deliberately blank: that run predates the client asking for usage in the stream, so it carries no prompt-token count of its own and no tokens-per-second figure is recorded for it. Whole-layer placement is what both engines ran (TensorSharp selects it with --tp 1); --tp 8 on this model means genuine tensor parallelism with every rank running every layer, which needs 41.7 GiB per rank and does not fit a 46 GB card — an accepted mode whose arithmetic and remedy the loader prints at load time, not a configuration that has been measured. Method, provenance and the rest of the matrix: docs/validation/cross-engine-2026-09/README.md.

MiniMax-H3: video and audio in one pass

MiniMax-H3 denoises the picture and a native 32 kHz stereo soundtrack together in one packed latent, so every figure below is for a run that produces both. Two machines have been measured against stable-diffusion.cpp on the same 22-frame, 8-step request, and they do not agree. Both are below, each labelled with its hardware; neither replaces the other, and an average of the two would describe no machine that exists.

Apple M5 Pro, ggml_metal

22 frames, 8 steps, identical seed, against stable-diffusion.cpp at its best-performing configuration:

Read the second row as the real one: faces need pixels, so 640×384 is the starting point and 256×256 is a corner the ratio flatters rather than a size to generate at. Both rows are the same 22-frame, 8-step request, and because the checkpoint is CFG-distilled those 8 steps are 8 DiT passes rather than 16. → MiniMax-H3 downloads and modes

RTX 3080 Laptop 16 GB, ggml_cuda

The same request on a deliberately memory-starved box — 16,384 MiB of VRAM and 31.7 GB of RAM against a 33.5 GB model set (DiT 10.64 GiB, Qwen3-VL-32B 16.97 GiB, video VAE 4.85 GiB, audio VAE 0.56 GiB, the identical files on both engines). 22 frames, 8 steps, identical seed, best of three, shipped build, against stable-diffusion.cpp 97d2990 with ggml 8e800ce rebuilt SD_CUDA=ON for arch 86. Here stable-diffusion.cpp takes the wall clock:

Peak VRAM on the 16,384 MiB card: TensorSharp 15,780 MiB, stable-diffusion.cpp 12,035 MiB. sd.cpp was given --auto-fit --stream-layers --diffusion-fa --rng cpu; its default --offload-to-cpu path cannot run this model on this machine at all, because it tries to pin 17.7 GB into 12.3 GB of free RAM.

Per denoise step TensorSharp is ahead on CUDA as well — 3.325 s against 3.338 s by the 8-vs-16-step slope, which is the measurement that isolates compute from setup, with 3.00–3.11 s seen directly per step. What it gives away is fixed setup cost, and roughly 3 s of the remaining 3.9 s is not inference at all: H.264 encoding, where sd.cpp writes MJPEG + PCM into an AVI, plus .NET process startup against a native binary.

What the last optimization pass moved

Same RTX 3080 Laptop, same 22-frame, 8-step workload, before and after:

Both changes aim at the same 16 GB ceiling. The finished denoiser now hands its device residency back before the video VAE loads, which takes peak VRAM during decode from 16,041 MiB to about 5,600 MiB and is worth 22 s at 640×384 — the finished denoiser and the video VAE come to 15.8 GB and do not both fit, and WDDM does not fail that allocation, it silently backs the overflow with host memory and runs the decode at PCIe speed. And the denoiser file is prefaulted in parallel with its own upload, taking the first denoise step from 14.87 s to ≈10.2 s: weights are bound as pointers into the mmapped GGUF, so without it the first host-to-device copy faults every page in from disk as it copies, at 0.91 GB/s against 5.97 GB/s from resident pageable memory. Output is byte-identical with the prefault on and off.

Which lever to pull, and in what order

Steps first. The default is 20 and 4–8 is the fast operating point — some chromatic fringing shows up around moving subjects at 4–8 and is gone by ~20. There is no guidance pass to cache or skip on top of that: the checkpoint ships CFG-distilled, --cfg 1.0 is required and anything higher is refused, which also leaves --negative-prompt and --cfg-cache-stride inert here — no unconditional pass runs for them to act on. Then resolution: --width / --height, rounded up to a multiple of 32, default 640×384. Then frame count: --video-frames, snapped up onto the 17k+5 grid (5, 22, 39, 56, 73, 90, 107 …) at a pinned 24 fps. Then the number of references, which past about four outweighs the step count entirely.

What a reference costs

The Ref2VA checkpoint takes up to nine references — stills, clips, standalone audio — and on the denoiser side they are cheap and completely predictable. On an RTX 3080 Laptop, 640×384 × 22 frames, one denoise step goes from 4.37 s with no references to 9.38 s with eight: about 626 ms per reference per step, flat from one reference to eight. The denoiser is not where the run goes.

Each reference adds roughly 250 vision placeholder tokens to the prompt, and the whole prompt is prefilled through all 50 layers of the Qwen3-VL-32B text encoder. Past about four references that pass, not the denoiser, is the run: at eight references the entire 8-step denoise costs ≈75 s against ≈447 s of text conditioning.

Numerics against the reference implementation

Each of the four networks is checked separately on the same inputs: text encoder cos 0.999999, one DiT step cos 0.998, video VAE encode and decode cos 1.000000, audio VAE decode cos 0.999995.

Wan video (video only): two independent speed levers

Video is the one workload where the wrong choice costs hours rather than seconds, so it is worth seeing the levers apart. Every row below is the same request — Apple M5 Pro (20-core GPU, 48 GB unified), ggml_metal, Wan2.2-TI2V-5B Q8_0, image-to-video at 1088×832 × 121 frames = 27,404 DiT tokens — and the rows differ only in the engine version and in which checkpoint --model pointed at.

The total decomposes into two multipliers that stack: ~1.7× per pass from engine work (F16 attention K/V plus the VAE convolution path) and 25× fewer passes from step distillation. The official Wan2.2-TI2V-5B recipe is 50 steps × 2 classifier-free-guidance passes = 100 DiT passes; a step-distilled checkpoint is trained guidance-free and costs 4. That is not a flag — it is a different --model file. TensorSharp detects it from the DiT file name (turbo, distill, lightning, lightx2v, fastwan, -dmd, or an explicit …-4steps-…), prints step-distilled checkpoint detected -> 4 steps, guidance off on load, and drops guidance automatically; --diffusion-steps / --cfg still override. → Wan downloads

Resolution and frame count vs. time

Same M5 Pro, same ggml_metal build, same Turbo checkpoint and the same source image. DiT self-attention costs O(tokens²) and the token count is latent_frames × (h/2) × (w/2), so frame area and frame count dominate everything else:

480p (≈0.4 MP) is a resolution Wan is trained at, so the 736×544 rows are in-distribution output rather than a degraded mode — that is the setting to reach for when a few minutes matters. Quality does fall off below ~0.3 MP, and the pipeline warns when width × height drops under 300,000 px.

Where the per-pass 1.7× comes from

A Wan VAE decode profile puts 44.4% of the graph in MUL_MAT and another 30.2% in IM2COL — 74.6% inside the convolution — which is why routing those convs through MPSGraph pays: per shape it reaches 6.2× (512→512 k3 at 320×240 t9), 8.6× (256→256 k3 at 640×480 t9), 10.4× (160→160 k3) and 13.9× (512→512 k1), because MPS reaches ~30 TFLOP/s where ggml's Metal GEMM gets ~4.9. Both knobs have an opt-out for A/B work: TS_WAN_DIT_KV_F16=0 and TS_WAN_VAE_MPS_CONV=0. Q8_0 attention K/V was measured slower than F16 (2,652 ms) and dequantizing the DiT from Q8_0 to F16 moved the matmuls by under 5%, so a higher-precision Wan quant buys essentially no speed.

Wan backends and the sd.cpp comparison

Reference timings on an RTX 2000 Ada (16 GB) with Wan2.1-1.3B F16 at 832×480, 33 frames, 30 UniPC steps — the official 480p recipe:

ggml_cuda is the backend to pick for Wan on NVIDIA; cpu / ggml_cpu run correctly but a 480p multi-second video takes tens of minutes there, and mlx is not a supported Wan backend at all. Against stable-diffusion.cpp (master-769, identical GGUFs and settings, 33-frame 480p) sampling is near parity — 281 s vs 258 s — but sd.cpp's VAE decode materializes ≈8 GB of 3D im2col and oversubscribes a 16 GB card into WDDM paging: 51 s vs 1,762 s, which makes TensorSharp 6.0× faster end to end on that run.

Testing

The build and test commands below require the .NET 10 SDK for your platform; verify the installation before running them.

Unit tests (xUnit)

InferenceWeb.Tests exercises in-process behavior that doesn't require a running server: managed quantized ops, direct CUDA and MLX backend kernels (when the hardware is available), paged-KV scheduling, batched-executor correctness, per-model batched-forward correctness against the legacy path, MTP / NextN speculative-decoding correctness, the pure-C# DeepSeek V4.1 executor's numerical gate against an independent PyTorch oracle, DiffusionGemma probes, codec round-trips, prompt rendering, and the server CLI options builder.

The V4.1 gate in that list is worth spelling out. InferenceWeb.Tests.Dsv41CpuExecutorTests holds DeepSeek4CpuExecutor — the 100% pure-C# executor that --backend cpu runs DeepSeek V4.1 Flash on, with no ggml, no native library and no GPU — to the independent PyTorch oracle eng/dsv41-reference.py at atol=rtol=2e-5 plus exact greedy-argmax agreement, over one-shot prefill, chunked prefill at 1/3/5/8 checked at every position, and reset. Two caveats travel with it: the tests return silently unless TS_DSV41_FIXTURE_DIR is set, and the fixture is small by design — a five-layer, 256-hidden, 16-token F32 synthetic model — so what it establishes is architectural agreement with the oracle, not CPU/CUDA parity on the real 246 GiB Q2_K weights. It is a correctness and portability path rather than a serving one, which is why it appears here and in no table above: no throughput, load time or resident footprint has been measured for a full V4.1 checkpoint on --backend cpu.

dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csproj

Server integration tests

Integration tests in TensorSharp.Server/testdata/ cover all three API styles (Web UI SSE, Ollama, OpenAI), multi-turn conversations, thinking mode, tool calling, structured outputs, queue-status compatibility, concurrent requests, and abort support. Architecture-specific features are auto-detected and skipped when the active model doesn't support them.

# Start TensorSharp.Server, then run:
python3 TensorSharp.Server/testdata/test_multiturn.py
# or
bash TensorSharp.Server/testdata/test_multiturn.sh

Inference matrix runner

TensorSharp.TestMatrix is the broader CLI-driven harness for long-running model × backend × feature × env-var coverage. It discovers GGUF files, filters unavailable backends and unsupported prompt types, runs baseline plus env-var sweep cells, writes one JSON result per cell, emits an aggregate Markdown report, and compares against per-host baselines.

dotnet build TensorSharp.TestMatrix/TensorSharp.TestMatrix.csproj -c Release
dotnet run --project TensorSharp.TestMatrix -c Release -- --dry-run