Compute Backends

A backend is selected with --backend (CLI and server) and decides which hardware runs the math. Every backend falls back to CPU for any unimplemented op, so output stays correct everywhere — backends differ only in speed.

Which backend should I use?

All backends in detail

GGML Metal, GGML CUDA & GGML Vulkan

The GPU paths built on a native C++ bridge that links ggml.

• GGML Metal (ggml_metal, macOS) — quantized weights are mapped zero-copy from the GGUF file into Metal command buffers via host-pointer buffers, so the resident set stays close to the on-disk model size.
• GGML CUDA (ggml_cuda, Windows/Linux + NVIDIA) — quantized weights are uploaded to device memory once at load time and the host copy is released afterwards.
• GGML Vulkan (ggml_vulkan, Windows/Linux + AMD/Intel/NVIDIA) — vendor-neutral GPU path for any GPU with a Vulkan 1.3 driver, using cooperative-matrix shaders (KHR coopmat / NV coopmat2) where the driver supports them. Weights are device-resident like GGML CUDA and the same fused whole-model decode/prefill graphs are used. On multi-GPU hosts (e.g. an integrated Intel GPU next to a discrete NVIDIA one), pick the device with --gpu-device N (or the TS_GGML_VULKAN_DEVICE env var) and list the visible devices with --list-gpus.

All three run native quantized matmul (Q4_K_M, Q8_0, …) without dequantizing to FP32, plus a native paged-attention kernel that drives ggml_flash_attn_ext.

Multi-GPU. GGML CUDA and GGML Vulkan also support tensor parallelism: --tp N gives each rank its own ggml backend, weight shards, and KV cache on its own GPU, driven concurrently by a rank worker pool, with cross-GPU AllReduce through ggml's collective (NCCL when the build finds it) or a host reduction for small payloads. Fused per-rank block graphs make --tp 2 decode faster than a single GPU on Gemma 4, and let models larger than one card's VRAM run entirely on GPUs.

MLX Metal

--backend mlx is a GPU-accelerated Apple-Silicon path built on mlx-c. It implements quantized ops (Q4_K_M, Q8_0, Q5_K, Q6_K, IQ2_XXS, IQ4_XS, IQ4_NL, MXFP4, …) without dequantizing to FP32, fused decode/prefill Metal kernels, compiled-graph kernels, async worker dispatch, batched MoE decode, and MoE expert offload. It pins the GGUF mmap in physical RAM via mlock(2) and derives allocator caps from the host's unified-memory capacity. Requires libmlxc (built locally or located via TENSORSHARP_MLX_LIBRARY / TENSORSHARP_MLX_LIBRARY_DIR).

Direct CUDA

--backend cuda is a pure-C# path using the CUDA Driver API, cuBLAS GEMM, and PTX kernels for common float32 ops (fill, unary/binary/ternary, activations, RMSNorm, softmax, RoPE/RoPEEx, SDPA, GQA prefill/decode, causal mask, gather/concat) plus native quantized matmul/get-rows for supported quant types. Unsupported ops route through CPU fallbacks while preserving tensor semantics. It is also the pure-C# backend where MTP speculative decoding is profitable.

It supports tensor parallelism: --tp N shards one model across N CUDA GPUs, and --tp-node-id / --tp-peers extend the group across machines. The GGML CUDA and GGML Vulkan backends shard too (see above); MLX and the CPU backends are single-device — on a multi-GPU host they pick one device (--gpu-device for Vulkan without --tp) rather than splitting the model. → Multi-GPU & Multi-Node

CPU backends

• Pure C# CPU (cpu) — portable inference with no native dependencies, using managed GEMM fast paths and fused SIMD kernels. Ideal for debugging and maximum portability.
• GGML CPU (ggml_cpu) — native GGML CPU kernels with quantized weights mapped zero-copy from the GGUF file. Faster than pure C# on most CPUs.

DeepSeek V4: dedicated whole-model executors

deepseek4 is the one architecture that does not use the generic per-op forward on any backend. Its 284B compressed-sparse-attention MoE stack runs through one of three dedicated whole-model executors instead:

• --backend cuda — a direct-CUDA engine independent of ggml. Quantized weights stream from the GGUF shards straight into per-device arenas.
• --backend ggml_cuda / ggml_vulkan — the native ggml executor: it loads the split GGUF itself, keeps all DSV4 KV state on-device, and runs each prefill/decode micro-batch as a single graph with a shape-signature cache so steady-state decode replays a captured CUDA graph.
• --backend cpu — a 100% pure-C# executor with no native dependencies, serving quantized weights straight from the memory-mapped GGUF shards.

All three layer-split the weights across every visible GPU (the CPU one streams them from the mapped shards), so a model far larger than one card still runs; --tp N or TS_DSV4_NGPU caps how many GPUs are used. Speculative decoding (DSpark) is available on the two GPU engines. → DeepSeek V4 downloads and commands

Building the native libraries

First install and verify the .NET 10 SDK for your platform. The native GGML library is built automatically on the first dotnet build. To build it manually or with CUDA:

cd TensorSharp.GGML.Native

# macOS (Metal)
bash build-macos.sh

# Linux — CPU only, force CUDA on, force Vulkan off
bash build-linux.sh
bash build-linux.sh --cuda
bash build-linux.sh --no-vulkan

# Windows — CPU only, force CUDA on, force Vulkan off
.\build-windows.ps1 --no-cuda
.\build-windows.ps1 --cuda
.\build-windows.ps1 --no-vulkan

The GGML Vulkan backend is enabled automatically when the machine has a Vulkan runtime (a loader such as vulkan-1.dll on Windows or libvulkan.so.1 on Linux, shipped by every recent GPU driver); opt out with --no-vulkan (or TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=OFF), and an explicit choice sticks across rebuilds via the CMake cache. On Windows, a LunarG Vulkan SDK is used when installed; without one the build auto-provisions a portable toolchain (Vulkan-Headers, a vulkan-1 import library generated from the system loader, glslc, SPIRV-Headers) into ExternalProjects/vulkan-toolchain/ via eng/fetch-vulkan-toolchain.ps1. On Linux, distro dev packages are used when present (apt install libvulkan-dev glslc spirv-headers); otherwise the build auto-provisions the missing pieces (Vulkan-Headers, glslc from the shaderc CI prebuilts, SPIRV-Headers) via eng/fetch-vulkan-toolchain.sh. A GPU driver with Vulkan 1.3 support is required at run time.

On Windows/Linux the script auto-detects the visible NVIDIA GPU compute capability and passes a narrow CMAKE_CUDA_ARCHITECTURES (e.g. 86-real on an RTX 3080), which cuts CUDA build time substantially. Override it explicitly:

TENSORSHARP_GGML_NATIVE_CUDA_ARCHITECTURES='86-real;89-real' bash build-linux.sh --cuda
bash build-linux.sh --cuda --cuda-arch='86-real;89-real'

You can also request CUDA from dotnet build directly:

TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.Cli/TensorSharp.Cli.csproj -c Release

MLX native library (macOS only)

The MLX backend depends on libmlxc. A helper script fetches and builds it:

bash TensorSharp.Backends.MLX/build-native-macos.sh

It writes the libraries into TensorSharp.Backends.MLX/Native/dist/. At run time the backend probes the application directory first; point it elsewhere with TENSORSHARP_MLX_LIBRARY or TENSORSHARP_MLX_LIBRARY_DIR.

Platform binary release status

The release workflow is intended to create the following self-contained archives after every required build job succeeds:

These are intended artifact names, not a claim that the files are currently downloadable. When present, -cuda archives still require an NVIDIA GPU and compatible driver; the macOS archive requires Apple Silicon.