Multi-GPU & Multi-Node

One model, many GPUs — and, when one machine is not enough, many machines. TensorSharp implements tensor parallelism in the Megatron-LM column/row-parallel pattern, and extends it across hosts with a peer-to-peer TCP mesh. On the architectures that shard no weights, the same --tp N flag runs a layer split instead — a capacity feature, not a speed one.

When do you need it?

• The model does not fit on one GPU. Each GPU holds only 1/N of the sharded weights, so a model that overflows a single card's VRAM fits comfortably across two or four.
• The model does not fit on one machine. Distributed TP pools the GPUs of several hosts into one logical group — 2 nodes × 4 GPUs is a global degree of 8.
• You want more memory bandwidth per token. Every GPU works on its own slice of every projection, so the per-layer matmuls are split rather than queued.

On the GGML backends, fused per-rank block graphs make --tp 2 decode faster than a single GPU on models that fit on one card too — 51.7 vs 37.3 tok/s on Gemma 4 E4B Q8_0 (2× RTX 2000 Ada). If you need more concurrent throughput rather than lower single-stream latency, reach for continuous batching first — it is on by default and costs nothing to try.

TP vs. the layer split — what a multi-GPU box actually does

There are two different ways a model can occupy more than one GPU, and only one of them is --tp.

• Tensor parallelism (--tp N) puts every layer on every rank and splits the weights within each layer, so a decode step reads 1/N of the bytes per device and the ranks all-reduce at each layer boundary. It is opt-in — nothing splits a tensor unless you ask — and most, though no longer all, of the architectures in the table below implement it.
• The layer split puts whole layers on different devices and runs them in sequence: device 0 evaluates layers 0..k, hands the hidden state to device 1, and so on. The cut points are not a naive n_layer/N — the loader measures each device's free VRAM and bin-packs the layers to balance the largest fraction-of-budget used, so an uneven set of cards still fills evenly. There are no collectives and no per-layer split, so it costs nothing on a slow interconnect, but only one GPU is busy at a time.

The layer split applies to four architectures, all of which run through their own whole-model executors. DeepSeek V4 Flash (deepseek4), DeepSeek V4.1 Flash (deepseek41) and GLM 5.x (glm-dsa, glm5next) do it by default, with no flag at all: they spread across every visible GPU because neither fits on one card. TS_DSV4_NGPU and TS_GLM_NGPU cap how many devices they use. Qwen 3.8 Flash Next (qwen4exp) is the one that asks: it stays on a single GPU until you pass --tp N, and what that runs is a layer split, not tensor parallelism — qwen4exp shards no weights, and this is the same (and only) multi-GPU mode llama.cpp offers the architecture, whose -sm row refuses to load it. For GLM-5.2, GLM-5.3 and GLM-5.3-Flash alike, omitting --tp keeps that default split, while --tp N on a GGML GPU backend selects native local/single-process tensor parallelism — GLM-5.3 (not Flash) is the same 79-block glm-dsa shape as 5.2, so it loads on the GLM-5.2 path with no new code and no new flag.

On every other architecture, running without --tp uses a single GPU. There is no automatic layer split on the generic per-op or fused-graph paths — a model that does not fit one card fails at load rather than being spread silently (the refusal that names the exact --n-cpu-moe N comes from the DeepSeek V4 and GLM 5.x whole-model loaders). An architecture that supports neither tensor parallelism nor a layer split now says so on stderr and runs on one GPU, instead of accepting --tp and quietly leaving the other cards idle.

So on a 3-GPU box, --backend ggml_cuda alone gives you all three GPUs on GLM 5.x and DeepSeek V4 (layer split) and one GPU on Gemma 4 and Qwen 3.8 Flash Next; adding --tp 3 switches GLM-5.2, GLM-5.3 and GLM-5.3-Flash to their native local/single-process tensor-parallel path, and caps DeepSeek V4's layer split at three devices (there --tp N is only a device count — the same thing TS_DSV4_NGPU sets), gives Gemma 4 all three GPUs by sharding inside its layers, and gives Qwen 3.8 Flash Next all three by splitting whole layers across them. On GLM-5.2 that switch is a downgrade in speed and buys only capacity; on plain GLM-5.3 it is an accepted mode that has never been run at any degree above one, and one whose replicated MLA and indexer caches multiply the KV footprint by N and shorten the context that fits; on Qwen 3.8 Flash Next the layer split is neither faster nor slower, and buys only capacity too — see Measured results.

How tensor parallelism works

Each transformer block is rewritten into a pair of complementary shardings, so exactly one collective is needed per block half:

1. Column-parallel projections

   QKV and gate/up split their output dimension — attention heads or the FFN intermediate size — across GPUs. No communication is needed: each rank simply produces its own slice of the activations.

2. Independent per-rank compute

   Attention (or the activation function) runs on each GPU over only the heads that GPU owns, against that GPU's own KV cache.

3. Row-parallel projections + AllReduce

   The output and down projections split their input dimension, so each rank computes a partial sum. One AllReduce adds the partials together and every rank ends the block holding the identical hidden state.

Norms, token embeddings, and the LM head are replicated on every rank rather than sharded — they are small, and replicating them removes a collective from the critical path.

Local TP — one process, several GPUs

Local TP lives in a single process. On the direct cuda backend one thread issues commands to all GPUs in sequence and CUDA streams provide the real parallelism; AllReduce runs as peer-to-peer device copies plus an element-wise add kernel. On the GGML backends a rank worker pool drives the GPUs concurrently — a GGML op submits and synchronizes in one call, so a sequential rank loop would run the cards strictly one after another.

# CLI — split the model across 2 GPUs
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2

# Same on the GGML CUDA backend
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend ggml_cuda --tp 2

# Pick which physical GPUs the ranks map to
TENSORSHARP_TP_DEVICES=0,2 dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model model.gguf --backend ggml_cuda --tp 2

# Server — same flag (TENSORSHARP_TP_DEGREE=2 also works)
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll \
    --model model.gguf --backend ggml_cuda --tp 2

Or put it in a config file:

{ "model": "model.gguf", "backend": "cuda", "tp": 2 }

Distributed TP — several machines

Each node runs its own process over its own local GPUs and connects to every other node over TCP with a length-prefixed framing protocol. AllReduce is hierarchical: reduce locally over P2P inside each node, exchange over TCP between node representatives, then broadcast back down — so only 1/tp_local of the data ever crosses the network.

# 2 nodes x 2 GPUs each = global TP degree 4.
# Node 0:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — same peer list, different node ID:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

The server can front such a cluster, but only as node 0 — the driver that owns sampling and serves HTTP. Every other node runs a TensorSharp.Cli worker with the same model, backend, and peer list. The TENSORSHARP_TP_* environment variables work in place of the flags.

# Node 0 — server / driver:
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model model.gguf --backend cuda \
    --tp 2 --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — CLI worker:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

Nodes are usually started by hand a few seconds or minutes apart, so a node that comes up first keeps retrying its outbound connections for up to 120 seconds instead of failing on the first refused connection. Once the mesh is complete each node prints [TcpCommunicator] Rank r/N connected to all peers.

Supported architectures

Nearly every autoregressive architecture in TensorSharp runs under TP; heterogeneous layers get their own sharding strategy. The ones whose --tp means something else are marked as such. This table describes local support; no GLM 5.x release claims distributed/cross-node TP — GLM-5.2, GLM-5.3 and GLM-5.3-Flash alike hard-refuse --tp-node-id/--tp-peers before the model is constructed.

Two MoE shardings are in play. Expert slicing (direct CUDA, and GPT OSS / Nemotron-H everywhere) gives each GPU 1/tp of every expert's weights with a replicated router, so load stays balanced whichever experts a token selects. Expert parallelism (Qwen 3.5/3.6 on GGML) partitions whole experts instead, which keeps each rank down to a single batched ggml_mul_mat_id dispatch per projection rather than a per-(token, expert) loop. Gemma 4 and GLM 5.x on GGML use a third variant: the Megatron split goes inside every expert (gate/up column-parallel, down row-parallel), so every rank still sees the full expert table and a token's selected ids stay globally valid — which is what ggml_mul_mat_id requires, since it will not accept the same expert id twice in one token's selection.

Requirements & constraints

• GPU backends only. TP runs on cuda, ggml_cuda, and ggml_vulkan. MLX and the CPU backends are single-device.
• Divisibility. numHeads, numKVHeads, and intermediateSize must all divide evenly by the TP degree. Quantized row-parallel splits additionally need ne0 divisible by tp × blockSize.
• Batched forward under TP is implemented for Mistral 3. MoE models (Gemma 4, Qwen 3.5/3.6, GPT OSS, Nemotron-H) fall back to per-sequence forward when TP is active, so continuous batching gives less of a lift there.
• Local degree ≤ visible GPUs. Requesting more ranks than the host has devices fails fast at startup.
• Multi-node keeps the per-op forward. The fused whole-model TP graphs are per-rank and single-process; a distributed run falls back to op-at-a-time execution.

Measured results

2× RTX 2000 Ada (16 GB each, PCIe, no NVLink), --backend ggml_cuda --tp 2, prefill 512 / decode 64, tokens per second:

^† Measured on 2× RTX PRO 4000 Blackwell (24 GB each, PCIe), not the RTX 2000 Ada pair the other rows use — the two hardware sets are not directly comparable to each other, only each row against its own 1-GPU column. Muse-Glimmer is the one model here that beats a single GPU on both phases (1.34× prefill, 1.57× decode); its Q8_0 has no 1-GPU number because 28.2 GB of weights does not fit on a 24 GB card at all.

That is not universal. On 3× RTX PRO 6000 (PCIe, no NVLink), GLM-5.2 UD-IQ2_XXS at prefill 2048 / decode 64 measures 915.9 / 43.9 tok/s on the plain layer split against 505.6 / 17.6 under --tp 3: each of the 78 layers needs two all-reduces of a [6144, n_tokens] hidden state, and on a PCIe host that costs more than the split saves. TP also holds a full-length cache on every rank, dropping the fitted context from 342,272 to 91,136 tokens, and it changes the reduction order — against the recorded llama.cpp goldens a 2-bit MoE reproduces 3 of 6 prompts under --tp 3 where the layer split reproduces 5 of 6. There, TP is a capacity feature rather than a latency one. How much TP buys depends on the interconnect and on how much of a layer can be split at all.

Decode — the memory-bound half TP is meant to help — reaches 1.39× a single GPU on Gemma 4 E4B and 1.06× on Qwen 3.5-9B, with both Gemma 4 models producing output byte-identical to their single-GPU runs. Prefill is compute-bound and pays the collectives, so it lands at or below the single-GPU figure for models that fit on one card. Qwen 3.5-35B-A3B does not fit a 16 GB card at all — TP is the only way to run it, and memory splits 9.4 + 8.0 GB across the pair.

The layer split, measured

A layer split shards no weights, so it should cost nothing and gain nothing — and that is what it measures. 2× A100-80GB, Qwen3.8-Flash-Next-UD-Q2_K_XL (73.4 GiB), --tp 2 against one GPU:

llama.cpp on the same box behaves the same way: -sm layer takes pp1536 / tg128 from 1094 / 61.2 on one GPU to 1200 / 61.5 on two — about 10% prefill, nothing on decode — and -sm row refuses to load this architecture at all. So the reason to pass --tp N here is that the weights, the caches and the context do not fit on one card, not throughput. Startup prints which mode ran and the per-GPU layer/byte split. TS_Q4E_LAYER_SPLIT=20,28 gives explicit layer counts per GPU (llama.cpp's --tensor-split in spirit) and throws rather than silently ignoring a value it cannot honour — useful, because the automatic balance prices weights and cannot see the vision tower, which loads later and lands on GPU 0.

CUDA graph capture under TP

A tensor-parallel token is dozens of small per-rank submissions, and replaying them is worth about 45% of decode throughput — so graph capture stays on under TP (disable with TS_GGML_TP_CUDA_GRAPHS=0). On 4×A40:

The collective transport is likewise chosen by measurement, not capability flags: at startup the group verifies that peer copies between the advertised device pairs actually deliver their bytes and that a real NCCL AllReduce completes, then picks the fastest transport that passes. Hosts that advertise peer access which never arrives (common on virtualized cloud instances) keep the NCCL collective with peer transport disabled rather than losing it — which matters past two GPUs, where the pinned-host pipeline does not apply and the alternative is reducing through host RAM at every layer boundary (4×A40, Qwen 3.5-9B Q8_0 decode: 53.5 → 75.1 tok/s).

Tuning & diagnostics

Local AllReduce prefers CUDA peer-to-peer DMA. At startup the group enables peer access for every pair that reports it, then runs a round-trip self-test — some topologies (L4 cards behind certain PCIe switches, IOMMU-enabled hosts, small BAR1 windows) claim peer access but silently transfer corrupt data, and any pair that fails is demoted to host staging permanently. Hardware with no peer access at all (A16 vGPU profiles, most consumer cards) stages through host memory from the start. The fallbacks are automatic; the switches below exist to force them for diagnosis.

Startup logs make the topology explicit: Tensor parallelism: N GPUs (<device names>) for the local group, a TP: P2P disabled… line or self-test warning when a pair is demoted, and one [TcpCommunicator] Rank r/N connected to all peers. per node.

Shared state across a cluster

When several server processes front the same workload, the server can optionally keep two pieces of state in Redis instead of in-process memory:

• KV cache tier — paged KV blocks are persisted so a prefix computed by one process can be reused by another (--redis-url / --paged-kv-redis-url, env TS_KV_CACHE_REDIS_URL; entries expire after TS_KV_CACHE_REDIS_TTL_MINUTES, default 24 h).
• Responses API store — the OpenAI Responses store becomes durable and shared instead of in-memory (TS_RESPONSES_STORE_REDIS_URL; --redis-url sets both at once).

# Both tiers on one Redis
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model model.gguf --backend cuda \
    --redis-url localhost:6379

# KV cache only, 12-hour TTL
dotnet TensorSharp.Server.Host/bin/TensorSharp.Server.Host.dll --model model.gguf --backend cuda \
    --paged-kv-redis-url localhost:6379 --paged-kv-redis-ttl 720

Troubleshooting