Multi-GPU & Multi-Node

One model, many GPUs — and, when one machine is not enough, many machines. TensorSharp implements tensor parallelism in the Megatron-LM column/row-parallel pattern, and extends it across hosts with a peer-to-peer TCP mesh.

When do you need it?

• The model does not fit on one GPU. Each GPU holds only 1/N of the sharded weights, so a model that overflows a single card's VRAM fits comfortably across two or four.
• The model does not fit on one machine. Distributed TP pools the GPUs of several hosts into one logical group — 2 nodes × 4 GPUs is a global degree of 8.
• You want more memory bandwidth per token. Every GPU works on its own slice of every projection, so the per-layer matmuls are split rather than queued.

On the GGML backends, fused per-rank block graphs make --tp 2 decode faster than a single GPU on models that fit on one card too — 51.7 vs 37.3 tok/s on Gemma 4 E4B Q8_0 (2× RTX 2000 Ada). If you need more concurrent throughput rather than lower single-stream latency, reach for continuous batching first — it is on by default and costs nothing to try.

How tensor parallelism works

Each transformer block is rewritten into a pair of complementary shardings, so exactly one collective is needed per block half:

1. Column-parallel projections

   QKV and gate/up split their output dimension — attention heads or the FFN intermediate size — across GPUs. No communication is needed: each rank simply produces its own slice of the activations.

2. Independent per-rank compute

   Attention (or the activation function) runs on each GPU over only the heads that GPU owns, against that GPU's own KV cache.

3. Row-parallel projections + AllReduce

   The output and down projections split their input dimension, so each rank computes a partial sum. One AllReduce adds the partials together and every rank ends the block holding the identical hidden state.

Norms, token embeddings, and the LM head are replicated on every rank rather than sharded — they are small, and replicating them removes a collective from the critical path.

Local TP — one process, several GPUs

Local TP lives in a single process. On the direct cuda backend one thread issues commands to all GPUs in sequence and CUDA streams provide the real parallelism; AllReduce runs as peer-to-peer device copies plus an element-wise add kernel. On the GGML backends a rank worker pool drives the GPUs concurrently — a GGML op submits and synchronizes in one call, so a sequential rank loop would run the cards strictly one after another.

# CLI — split the model across 2 GPUs
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2

# Same on the GGML CUDA backend
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend ggml_cuda --tp 2

# Pick which physical GPUs the ranks map to
TENSORSHARP_TP_DEVICES=0,2 dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll \
    --model model.gguf --backend ggml_cuda --tp 2

# Server — same flag (TENSORSHARP_TP_DEGREE=2 also works)
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll \
    --model model.gguf --backend ggml_cuda --tp 2

Or put it in a config file:

{ "model": "model.gguf", "backend": "cuda", "tp": 2 }

Distributed TP — several machines

Each node runs its own process over its own local GPUs and connects to every other node over TCP with a length-prefixed framing protocol. AllReduce is hierarchical: reduce locally over P2P inside each node, exchange over TCP between node representatives, then broadcast back down — so only 1/tp_local of the data ever crosses the network.

# 2 nodes x 2 GPUs each = global TP degree 4.
# Node 0:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — same peer list, different node ID:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

The server can front such a cluster, but only as node 0 — the driver that owns sampling and serves HTTP. Every other node runs a TensorSharp.Cli worker with the same model, backend, and peer list. The TENSORSHARP_TP_* environment variables work in place of the flags.

# Node 0 — server / driver:
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --tp 2 --tp-node-id 0 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

# Node 1 — CLI worker:
dotnet TensorSharp.Cli/bin/TensorSharp.Cli.dll --model model.gguf --backend cuda --tp 2 \
    --tp-node-id 1 --tp-peers "192.168.1.10:9500,192.168.1.11:9500"

Nodes are usually started by hand a few seconds or minutes apart, so a node that comes up first keeps retrying its outbound connections for up to 120 seconds instead of failing on the first refused connection. Once the mesh is complete each node prints [TcpCommunicator] Rank r/N connected to all peers.

Supported architectures

Every autoregressive architecture in TensorSharp runs under TP; heterogeneous layers get their own sharding strategy.

Two MoE shardings are in play. Expert slicing (direct CUDA, and GPT OSS / Nemotron-H everywhere) gives each GPU 1/tp of every expert's weights with a replicated router, so load stays balanced whichever experts a token selects. Expert parallelism (Qwen 3.5/3.6 on GGML) partitions whole experts instead, which keeps each rank down to a single batched ggml_mul_mat_id dispatch per projection rather than a per-(token, expert) loop.

Requirements & constraints

• GPU backends only. TP runs on cuda, ggml_cuda, and ggml_vulkan. MLX and the CPU backends are single-device.
• Divisibility. numHeads, numKVHeads, and intermediateSize must all divide evenly by the TP degree. Quantized row-parallel splits additionally need ne0 divisible by tp × blockSize.
• Batched forward under TP is implemented for Qwen 3 and Mistral 3. MoE models (Gemma 4, Qwen 3.5/3.6, GPT OSS, Nemotron-H) fall back to per-sequence forward when TP is active, so continuous batching gives less of a lift there.
• Local degree ≤ visible GPUs. Requesting more ranks than the host has devices fails fast at startup.
• Multi-node keeps the per-op forward. The fused whole-model TP graphs are per-rank and single-process; a distributed run falls back to op-at-a-time execution.

Measured results

2× RTX 2000 Ada (16 GB each, PCIe, no NVLink), --backend ggml_cuda --tp 2, prefill 512 / decode 64, tokens per second:

Decode — the memory-bound half TP is meant to help — reaches 1.39× a single GPU on Gemma 4 E4B and 1.06× on Qwen 3.5-9B, with both Gemma 4 models producing output byte-identical to their single-GPU runs. Prefill is compute-bound and pays the collectives, so it lands at or below the single-GPU figure for models that fit on one card. Qwen 3.5-35B-A3B does not fit a 16 GB card at all — TP is the only way to run it, and memory splits 9.4 + 8.0 GB across the pair.

Tuning & diagnostics

Local AllReduce prefers CUDA peer-to-peer DMA. At startup the group enables peer access for every pair that reports it, then runs a round-trip self-test — some topologies (L4 cards behind certain PCIe switches, IOMMU-enabled hosts, small BAR1 windows) claim peer access but silently transfer corrupt data, and any pair that fails is demoted to host staging permanently. Hardware with no peer access at all (A16 vGPU profiles, most consumer cards) stages through host memory from the start. The fallbacks are automatic; the switches below exist to force them for diagnosis.

Startup logs make the topology explicit: Tensor parallelism: N GPUs (<device names>) for the local group, a TP: P2P disabled… line or self-test warning when a pair is demoted, and one [TcpCommunicator] Rank r/N connected to all peers. per node.

Shared state across a cluster

When several server processes front the same workload, the server can optionally keep two pieces of state in Redis instead of in-process memory:

• KV cache tier — paged KV blocks are persisted so a prefix computed by one process can be reused by another (--redis-url / --paged-kv-redis-url, env TS_KV_CACHE_REDIS_URL; entries expire after TS_KV_CACHE_REDIS_TTL_MINUTES, default 24 h).
• Responses API store — the OpenAI Responses store becomes durable and shared instead of in-memory (TS_RESPONSES_STORE_REDIS_URL; --redis-url sets both at once).

# Both tiers on one Redis
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --redis-url localhost:6379

# KV cache only, 12-hour TTL
dotnet TensorSharp.Server/bin/TensorSharp.Server.dll --model model.gguf --backend cuda \
    --paged-kv-redis-url localhost:6379 --paged-kv-redis-ttl 720

Troubleshooting