Multi-GPU & Multi-Node

--tp N selects tensor parallelism, sharding weights within layers. --layer-split N selects whole-layer placement on supported architectures. The flags are mutually exclusive; unsupported requests fail at startup. Layer split is local to one node. Only supported tensor-parallel architectures can use --tp-node-id / --tp-peers across nodes. Migrate old layer-placement commands from --tp N to --layer-split N (environment: TENSORSHARP_LAYER_SPLIT_DEGREE=N).

When do you need it?

• The model does not fit on one GPU. Each GPU holds only 1/N of the sharded weights, so a model that overflows a single card's VRAM fits comfortably across two or four.
• The model does not fit on one machine. Distributed TP pools the GPUs of several hosts into one logical group — 2 nodes × 4 GPUs is a global degree of 8.
• You want more memory bandwidth per token. Every GPU works on its own slice of every projection, so the per-layer matmuls are split rather than queued.

On the GGML backends, fused per-rank block graphs make --tp 2 decode faster than a single GPU on models that fit on one card too — 51.7 vs 37.3 tok/s on Gemma 4 E4B Q8_0 (2× RTX 2000 Ada). If you need more concurrent throughput rather than lower single-stream latency, reach for continuous batching first — it is on by default and costs nothing to try.

Choose tensor parallelism or layer split

Tensor parallelism (--tp N) splits projections inside each layer across N GPUs. The ranks compute concurrently and combine partial results with collectives. Layer split (--layer-split N) assigns whole layers to N local GPUs and transfers hidden states at device boundaries. Layer split increases capacity; it does not guarantee a throughput improvement.

Qwen 3.8 Flash Next supports local --tp N on ggml_cuda for eligible FFN quantizations, and --layer-split N on GGML CUDA/Vulkan. DeepSeek V4.1 supports routed-MoE TP on GGML CUDA (degrees 2–8), while attention and caches retain layer placement. DeepSeek V4 and GLM 5.x also support local whole-layer placement; GLM has local native TP. The flags are mutually exclusive, model/backend checks reject unsupported modes, and these whole-model paths do not support multi-node groups.

# Whole layers on two local GPUs; the server accepts the same option
TensorSharp.Cli/bin/TensorSharp.Cli --model qwen38-flash-next.gguf \
    --backend ggml_cuda --layer-split 2

Config equivalent: { "layer-split": 2 }. Environment equivalent: TENSORSHARP_LAYER_SPLIT_DEGREE=2. Do not combine it with tensor-parallel or distributed settings; multi-node layer split is not implemented.

How tensor parallelism works

Each transformer block is rewritten into a pair of complementary shardings, so exactly one collective is needed per block half:

1. Column-parallel projections

   QKV and gate/up split their output dimension — attention heads or the FFN intermediate size — across GPUs. No communication is needed: each rank simply produces its own slice of the activations.

2. Independent per-rank compute

   Attention (or the activation function) runs on each GPU over only the heads that GPU owns, against that GPU's own KV cache.

3. Row-parallel projections + AllReduce

   The output and down projections split their input dimension, so each rank computes a partial sum. One AllReduce adds the partials together and every rank ends the block holding the identical hidden state.

Norms, token embeddings, and the LM head are replicated on every rank rather than sharded — they are small, and replicating them removes a collective from the critical path.

Local TP — one process, several GPUs

Local TP lives in a single process. On the direct cuda backend one thread issues commands to all GPUs in sequence and CUDA streams provide the real parallelism; AllReduce runs as peer-to-peer device copies plus an element-wise add kernel. On the GGML backends a rank worker pool drives the GPUs concurrently — a GGML op submits and synchronizes in one call, so a sequential rank loop would run the cards strictly one after another.

# CLI — split the model across 2 GPUs
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2

# Same on the GGML CUDA backend
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend ggml_cuda --tp 2

# Pick which physical GPUs the ranks map to
TENSORSHARP_TP_DEVICES=0,2 dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model model.gguf --backend ggml_cuda --tp 2

# Server — same flag (TENSORSHARP_TP_DEGREE=2 also works)
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model model.gguf --backend ggml_cuda --tp 2

Or put it in a config file:

{ "model": "model.gguf", "backend": "cuda", "tp": 2 }

Distributed TP — several machines

Each node runs its own process over its own local GPUs and connects to every other node over TCP with a length-prefixed framing protocol. AllReduce is hierarchical: reduce locally over P2P inside each node, exchange over TCP between node representatives, then broadcast back down — so only 1/tp_local of the data ever crosses the network.

# 2 nodes x 2 GPUs each = global TP degree 4.
# Node 0:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — same peer list, different node ID:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

The server can front such a cluster, but only as node 0 — the driver that owns sampling and serves HTTP. Every other node runs a TensorSharp.Cli worker with the same model, backend, and peer list. The TENSORSHARP_TP_* environment variables work in place of the flags.

# Node 0 — server / driver:
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model model.gguf --backend cuda \
    --tp 2 --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — CLI worker:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

Nodes are usually started by hand a few seconds or minutes apart, so a node that comes up first keeps retrying its outbound connections for up to 120 seconds instead of failing on the first refused connection. Once the mesh is complete each node prints [TcpCommunicator] Rank r/N connected to all peers.

Supported architectures

Nearly every autoregressive architecture in TensorSharp runs under TP, and so does Qwen-Image-2.1's diffusion transformer; heterogeneous layers get their own sharding strategy. Layer-only architectures are marked separately and require --layer-split. This table describes local support; no GLM 5.x release claims distributed/cross-node TP — GLM-5.2, GLM-5.3 and GLM-5.3-Flash alike hard-refuse --tp-node-id/--tp-peers before the model is constructed.

Two MoE shardings are in play. Expert slicing (direct CUDA, and GPT OSS / Nemotron-H everywhere) gives each GPU 1/tp of every expert's weights with a replicated router, so load stays balanced whichever experts a token selects. Expert parallelism (Qwen 3.5/3.6 on GGML) partitions whole experts instead, which keeps each rank down to a single batched ggml_mul_mat_id dispatch per projection rather than a per-(token, expert) loop. Gemma 4 and GLM 5.x on GGML use a third variant: the Megatron split goes inside every expert (gate/up column-parallel, down row-parallel), so every rank still sees the full expert table and a token's selected ids stay globally valid — which is what ggml_mul_mat_id requires, since it will not accept the same expert id twice in one token's selection.

Requirements & constraints

• GPU backends only. TP runs on cuda, ggml_cuda, and ggml_vulkan. MLX and the CPU backends are single-device.
• Divisibility. numHeads, numKVHeads, and intermediateSize must all divide evenly by the TP degree. Quantized row-parallel splits additionally need ne0 divisible by tp × blockSize.
• Batched forward under TP is implemented for Mistral 3. MoE models (Gemma 4, Qwen 3.5/3.6, GPT OSS, Nemotron-H) fall back to per-sequence forward when TP is active, so continuous batching gives less of a lift there.
• Local degree ≤ visible GPUs. Requesting more ranks than the host has devices fails fast at startup.
• Multi-node keeps the per-op forward. The fused whole-model TP graphs are per-rank and single-process; a distributed run falls back to op-at-a-time execution.

Measured results

2× RTX 2000 Ada (16 GB each, PCIe, no NVLink), --backend ggml_cuda --tp 2, prefill 512 / decode 64, tokens per second:

^† Measured on 2× RTX PRO 4000 Blackwell (24 GB each, PCIe), not the RTX 2000 Ada pair the other rows use — the two hardware sets are not directly comparable to each other, only each row against its own 1-GPU column. Muse-Glimmer is the one model here that beats a single GPU on both phases (1.34× prefill, 1.57× decode); its Q8_0 has no 1-GPU number because 28.2 GB of weights does not fit on a 24 GB card at all.

That is not universal. On 3× RTX PRO 6000 (PCIe, no NVLink), GLM-5.2 UD-IQ2_XXS at prefill 2048 / decode 64 measures 915.9 / 43.9 tok/s on the plain layer split against 505.6 / 17.6 under --tp 3: each of the 78 layers needs two all-reduces of a [6144, n_tokens] hidden state, and on a PCIe host that costs more than the split saves. TP also holds a full-length cache on every rank, dropping the fitted context from 342,272 to 91,136 tokens, and it changes the reduction order — against the recorded llama.cpp goldens a 2-bit MoE reproduces 3 of 6 prompts under --tp 3 where the layer split reproduces 5 of 6. There, TP is a capacity feature rather than a latency one. How much TP buys depends on the interconnect and on how much of a layer can be split at all.

Decode — the memory-bound half TP is meant to help — reaches 1.39× a single GPU on Gemma 4 E4B and 1.06× on Qwen 3.5-9B, with both Gemma 4 models producing output byte-identical to their single-GPU runs. Prefill is compute-bound and pays the collectives, so it lands at or below the single-GPU figure for models that fit on one card. Qwen 3.5-35B-A3B does not fit a 16 GB card at all — TP is the only way to run it, and memory splits 9.4 + 8.0 GB across the pair.

The layer split, measured

A layer split shards no weights, so it should cost nothing and gain nothing — and that is what it measures. 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB), --layer-split 2 against one GPU:

llama.cpp on the same box behaves the same way: -sm layer takes pp1536 / tg128 from 1094 / 61.2 on one GPU to 1200 / 61.5 on two — about 10% prefill, nothing on decode — and -sm row refuses to load this architecture at all. So the reason to pass --layer-split N here is that the weights, the caches and the context do not fit on one card, not throughput. Startup prints which mode ran and the per-GPU layer/byte split. TS_Q4E_LAYER_SPLIT=20,28 gives explicit layer counts per GPU (llama.cpp's --tensor-split in spirit) and throws rather than silently ignoring a value it cannot honour — useful, because the automatic balance prices weights and cannot see the vision tower, which loads later and lands on GPU 0.

Qwen-Image-2.1's diffusion transformer under TP

Two NVIDIA A40s (46 GB each) in different CPU sockets, where peer access is not functional and NCCL used its shared-memory transport. Prefix cache on, step times exclude step 1:

On Vulkan the reduction goes through host memory and outweighs the split, so --tp is slower there on this hardware. TP changes the order in which each block's partial sums are added, so images are not bit-identical to one GPU (34.2–51.5 dB PSNR against the single-GPU image on CUDA): composition and quality hold, but details can move. Detail: the repository's docs/models/qwenimage21.md card.

CUDA graph capture under TP

A tensor-parallel token is dozens of small per-rank submissions, and replaying them is worth about 45% of decode throughput — so graph capture stays on under TP (disable with TS_GGML_TP_CUDA_GRAPHS=0). On 4×A40:

The collective transport is likewise chosen by measurement, not capability flags: at startup the group verifies that peer copies between the advertised device pairs actually deliver their bytes and that a real NCCL AllReduce completes, then picks the fastest transport that passes. Hosts that advertise peer access which never arrives (common on virtualized cloud instances) keep the NCCL collective with peer transport disabled rather than losing it — which matters past two GPUs, where the pinned-host pipeline does not apply and the alternative is reducing through host RAM at every layer boundary (4×A40, Qwen 3.5-9B Q8_0 decode: 53.5 → 75.1 tok/s).

Tuning & diagnostics

Local AllReduce prefers CUDA peer-to-peer DMA. At startup the group enables peer access for every pair that reports it, then runs a round-trip self-test — some topologies (L4 cards behind certain PCIe switches, IOMMU-enabled hosts, small BAR1 windows) claim peer access but silently transfer corrupt data, and any pair that fails is demoted to host staging permanently. Hardware with no peer access at all (A16 vGPU profiles, most consumer cards) stages through host memory from the start. The fallbacks are automatic; the switches below exist to force them for diagnosis.

Startup logs make the topology explicit: Tensor parallelism: N GPUs (<device names>) for the local group, a TP: P2P disabled… line or self-test warning when a pair is demoted, and one [TcpCommunicator] Rank r/N connected to all peers. per node.

Shared state across a cluster

When several server processes front the same workload, one piece of state can live in Redis instead of in-process memory:

• Responses API store — the OpenAI Responses store becomes durable and shared instead of in-memory (--redis-url, env TS_RESPONSES_STORE_REDIS_URL).
• No shared KV cache tier — there is no cross-process KV reuse through Redis: Redis backs only the Responses API store. Each process keeps its own radix prefix cache, and its prefix checkpoints persist beside its own binary under prefix-cache/.

# Responses store on one shared Redis
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model model.gguf --backend cuda \
    --redis-url localhost:6379

Troubleshooting