Benchmarks & Testing

Recent recorded DeepSeek V4.1 Flash 1017, Muse Spark and MiniMax-M3.1-Flash comparisons, with exact workload and quality limits, plus the correctness test harness.

Latest recorded text comparisons

These selected results cover DeepSeek V4.1 Flash 1017 on five A40 GPUs, Muse Spark on ten A40 GPUs and MiniMax-M3.1-Flash on eight A40 GPUs. Each row keeps its model, quantization, placement, workload and reference mode. The best recorded TensorSharp mode differs by metric; the rows do not form a combined score or establish a speedup for other hardware or models. The DeepSeek and Muse Spark benchmarks were not rerun for this documentation update; the MiniMax-M3.1-Flash runs are from 2026-10-24.

DeepSeek V4.1 Flash 1017: exact and approximate prefill

Recorded 2026-10-22–23: five NVIDIA A40 GPUs (46 GB each), unsloth/DeepSeek-V4.1-Flash-1017-GGUF UD-IQ2_XXS, six shards, Hugging Face revision 60ad3651. TensorSharp uses unchanged ggml da8e2a5 and --tp 5; the reference is the Unsloth llama.cpp checkout be836572 with layer split and its own in-tree ggml. Context is 32,768 tokens, concurrency 1, 256 generated tokens, greedy sampling and thinking off. Values are medians of three runs after a warmup.

The default YOCO replay is approximate. Exact prefill uses TS_DSV41_YOCO_KEEP=0. On the recorded teacher-forced wikitext-2 check, perplexity is 2.4811 exact / 2.4978 default approximate / 2.4697 llama.cpp (lower is better). Approximate throughput must be read with that quality cost; it is not an exact-execution speedup.

Prompt / metricTensorSharp TP5 exactTensorSharp TP5 default approximate YOCOllama.cpp layer splitExact / referenceApproximate / reference
~1.8K tokens, prefill tok/s563.9960.5174.63.23×5.50×
~7.3K tokens, prefill tok/s549.71,000.7225.72.44×4.43×

These rates use the engines' internal prefill timers, not streamed TTFT or client wall time. The reference measurements came from a different session; no confidence interval or claim of an interleaved controlled experiment is made. The 5.50× cell is only the ~1.8K-token, single-request, default-approximate TP5 workload above. Model layout, launch settings, numerical checks and build/provenance limits: DeepSeek V4.1 model card. Choose a supported path →

Muse Spark 816B on ten A40 GPUs

Report refreshed 2026-10-22; the measurement date is not recorded. Muse-Spark-Open-816B-A42B, UD-IQ3_XXS, eight shards, all experts in GPU memory, ten A40 46 GB GPUs, ggml_cuda, context 32,768, F16 KV, flash attention and thinking off. TensorSharp's recorded working-tree build uses unchanged upstream ggml da8e2a5; llama.cpp is unmodified 7170cad, with its own in-tree ggml. Layer split is compared with -sm layer, tensor parallelism with -sm tensor.

WorkloadMatching placementTensorSharpllama.cppRatio
Decode, one request, 167-token prompt / 256 output tokensLayer split16.69 tok/s16.79 tok/s0.99×
Prefill, 4,075-token prompt, one request / up to 32 output tokensLayer split999.7 tok/s483.2 tok/s2.07×
Prefill, 15,767-token prompt, one request / up to 32 output tokensLayer split1,346.8 tok/s826.1 tok/s1.63×
Decode, one request, 167-token prompt / 256 output tokensTensor parallelism22.63 tok/s16.19 tok/s1.40×
Aggregate output, four concurrent requests / 256 tokens eachTensor parallelism45.73 tok/s24.49 tok/s1.87×

Muse Spark prefill is uncached prompt tokens divided by streamed TTFT, unlike DeepSeek's internal-timer metric above. Layer split gives the strongest recorded prefill, while tensor parallelism gives the strongest recorded generation throughput. Ratios above 1 mean higher throughput; the 0.99× layer-split decode row is retained. TensorSharp and llama.cpp layer-split timings are medians of three repeats; llama.cpp tensor-split rows use medians of two. The reference's short and 4K tensor-split rows came from an earlier session of the same build. The teacher-forced layer-split mean KL divergence from llama.cpp is 0.0026; integration fixtures do not establish general model quality.

One host, no working P2P or NVLink: TensorSharp uses NCCL through host memory, while llama.cpp was built without NCCL because that build aborted on ten GPUs. Transport conditions are therefore material. The model files sit on a network filesystem. Images, speculation, CPU expert offload, contexts above 16K, other GPU counts and non-CUDA backends were not measured. The report does not freeze the final TensorSharp commit or commit complete raw records. This is a recorded working-tree comparison, not a benchmark of this documentation commit. Full setup, per-scenario results and limitations →

MiniMax-M3.1-Flash on eight A40 GPUs

Measured 2026-10-24. Unsloth MiniMax-M3.1-Flash UD-Q4_K_XL, seven shards / 232.3 GiB, eight A40 46 GB GPUs, ggml_cuda, layer split unless noted. TensorSharp ran on unchanged upstream ggml da8e2a5; the reference is llama.cpp from unslothai's minimax-m31 branch (ef8f94cc, PR #320). Same box, one engine at a time with the GPUs held exclusively, three repeats, medians.

WorkloadMethodTensorSharpllama.cppRatio
Prefill, 512 / 8,192 / 32,768 / 65,536 tokensExecutor bench vs llama-bench -fa 1448 / 1,699 / 2,449 / 2,567 tok/s344 / 1,312 / 681 / 262 tok/s1.30× / 1.29× / 3.60× / 9.80×
Decode, 128 tokens at depth 0 / 2,048 / 8,192 / 32,768Executor bench vs llama-bench -fa 124.8 / 23.9 / 23.8 / 23.4 tok/s22.5 / 19.6 / 19.1 / 17.5 tok/s1.10× / 1.22× / 1.25× / 1.34×
Prefill, 4K / 16K / 32K / 64K-token promptServer scenarios, one sequence1,263 / 1,478 / 1,544 / 1,545 tok/s283 / 273 / 240 / 211 tok/s4.46× / 5.41× / 6.43× / 7.32×
Decode after those promptsServer scenarios, one sequence23.9 / 23.7 / 23.5 / 23.0 tok/s19.2 / 18.5 / 17.7 / 16.1 tok/s1.24× / 1.28× / 1.33× / 1.43×
Time to first token, short chatServer scenarios0.41 s0.39 s0.95×
Aggregate output, 4 / 8 concurrent requestsServer scenarios42.1 / 53.7 tok/s31.8 / 40.6 tok/s1.32× / 1.32×
DSpark speculative decode, mean of code / prose / copyServer scenarios, default gate vs draft-dspark32.3 tok/s25.8 tok/s1.25×
Decode at depth 0, --tp 8TP bench35.7 tok/sno tensor split for this model—

Server prefill is uncached prompt tokens divided by streamed time to first token; llama-server ran with -np 1 --no-kv-unified (without it MiniMax Sparse Attention runs dense), engines rotated ABBA (the page cache was not evicted between launches, so load times are not compared). Each llama-bench prefill cell is the faster of -ub 512 and -ub 2048, and its 64K cell is a single run. Short-chat and concurrent time to first token are slightly behind llama.cpp. Without the drafter both engines decoded those DSpark prompts at 22.9 and 20.5 tok/s. --tp 8 drops to 26.0–26.5 tok/s past depth 2,048 and caps the context at 45,824 tokens, because attention and the caches are mirrored on every GPU; its figure comes from the tensor-parallel branch's build before integration.

Limits: uncommitted working-tree builds rather than a frozen commit; one host and one checkpoint, no confidence intervals. Video, other GPU counts and backends, and full-checkpoint perplexity or KL divergence were not measured (MoE CPU offload was: 1.30× decode and 1.62× 4K prefill, two repeats); logit parity with llama.cpp was checked on synthetic fixtures only. The raw records are local validation evidence on the measurement host, not committed. Every cell, method and limit →

Other model families have their own capabilities and engineering notes: text models, image models and video models. The selected text comparisons above make no performance claim for those workloads.

Testing

The build and test commands below require the .NET 10 SDK for your platform; verify the installation before running them.

Unit tests (xUnit)

InferenceWeb.Tests exercises in-process behavior that doesn't require a running server: managed quantized ops, direct CUDA and MLX backend kernels (when the hardware is available), paged-KV scheduling, batched-executor correctness, per-model batched-forward correctness against the single-sequence path, MTP / NextN speculative-decoding correctness, the pure-C# DeepSeek V4.1 executor's numerical gate against an independent PyTorch oracle, DiffusionGemma probes, codec round-trips, prompt rendering, and the server CLI options builder.

The V4.1 gate in that list is worth spelling out. InferenceWeb.Tests.Dsv41CpuExecutorTests holds DeepSeek4CpuExecutor — the 100% pure-C# executor that --backend cpu runs DeepSeek V4.1 Flash on, with no ggml, no native library and no GPU — to the independent PyTorch oracle eng/dsv41-reference.py at atol=rtol=2e-5 plus exact greedy-argmax agreement, over one-shot prefill, chunked prefill at 1/3/5/8 checked at every position, and reset. Two caveats travel with it: fixture-dependent checks require TS_DSV41_FIXTURE_DIR and unavailable scenarios are explicit skips, and the fixture is small by design — a five-layer, 256-hidden, 16-token F32 synthetic model — so what it establishes is architectural agreement with the oracle, not CPU/CUDA parity on the real 246 GiB Q2_K weights. It is a correctness and portability path rather than a serving one, which is why it appears here and in no table above: no throughput, load time or resident footprint has been measured for a full V4.1 checkpoint on --backend cpu.

dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csproj

Pull-request CI runs the environment-independent lane of that suite on GitHub-hosted x64 (ubuntu-latest) and ARM64 (ubuntu-24.04-arm) runners. It forces the GPU native builds off (CUDA, Vulkan and cuDNN), pins TS_TEST_GGML_BACKEND=cpu, first runs the GPU-free GB10 container-contract check eng/tests/gb10-container.py, and needs no GPU and no model downloads. The same lane runs locally with:

dotnet test InferenceWeb.Tests/InferenceWeb.Tests.csproj --filter "Category!=Bench&Requires!=Cuda&Requires!=Mlx&Requires!=Models"

Server integration tests

Integration tests in TensorSharp.Server.Host/testdata/ cover all three API styles (Web UI SSE, Ollama, OpenAI), multi-turn conversations, thinking mode, tool calling, structured outputs, queue-status compatibility, concurrent requests, and abort support. Architecture-specific features are auto-detected and skipped when the active model doesn't support them.

# Start TensorSharp.Server.Host, then run:
python3 TensorSharp.Server.Host/testdata/test_multiturn.py
# or
bash TensorSharp.Server.Host/testdata/test_multiturn.sh

Inference matrix runner

TensorSharp.TestMatrix is the broader CLI-driven harness for long-running model × backend × feature × env-var coverage. It discovers GGUF files, filters unavailable backends and unsupported prompt types, runs baseline plus env-var sweep cells, writes one JSON result per cell, emits an aggregate Markdown report, and compares against per-host baselines.

dotnet build TensorSharp.TestMatrix/TensorSharp.TestMatrix.csproj -c Release
dotnet run --project TensorSharp.TestMatrix -c Release -- --dry-run
📊

Benchmark numbers depend heavily on hardware, model, quantization, and KV-cache dtype. Each comparison above has its own conditions and evidence limits; historical snapshots do not establish current performance. Run the built-in benchmark on your own hardware for the numbers that matter to you.