Using TensorSharp from C#

TensorSharp is a real .NET library, not just an executable. Reference its projects from a source checkout and drive inference directly from your own code when you do not want an HTTP hop.

Project and package boundaries

The repository is split along publishable package boundaries. The names below are project/package IDs, but the current Runtime/Models/Backends/CLI/Server packages are not available on NuGet.org.

For a typical embedding scenario, reference TensorSharp.Models.csproj; it already references Core, Runtime, and the backend projects. From a sibling application beside the TensorSharp checkout:

dotnet add reference ../TensorSharp/TensorSharp.Models/TensorSharp.Models.csproj

Minimal example — load, generate, decode

Every model is loaded the same way: ModelBase.Create() reads the GGUF metadata and instantiates the right architecture. From there you tokenize, run forward passes, sample, and decode.

using System;
using System.Collections.Generic;
using System.Linq;
using TensorSharp.Models;
using TensorSharp.Runtime;

// 1. Load any supported GGUF — the architecture is detected from metadata.
//    Pick the BackendType your build supports: GgmlCuda, GgmlMetal, GgmlVulkan, or GgmlCpu.
using var model = ModelBase.Create("gemma-4-E4B-it-Q8_0.gguf", BackendType.GgmlCuda);

// 2. Configure sampling (defaults match Ollama: temp 0.8, top_k 40, top_p 0.9).
var sampling = new SamplingConfig { Temperature = 0.7f, TopP = 0.9f, TopK = 40 };

// 3. Tokenize the prompt.
var tokens = model.Tokenizer
    .Encode("Explain mixture-of-experts in one sentence.", addSpecial: true)
    .ToList();
var generated = new List<int>();

// 4. Prefill the complete prompt once.
float[] logits = model.Forward(tokens.ToArray());

// 5. Decode one new token at a time. Forward() owns the KV-cache position,
// so after prefill pass only the newly sampled token, not the full history.
for (int step = 0; step < 200; step++)
{
    int next = model.Sample(logits, sampling, generated); // applies penalties + sampling
    if (model.Tokenizer.IsEos(next)) break;
    generated.Add(next);
    logits = model.Forward(new[] { next });
}

// 6. Detokenize the result.
Console.WriteLine(model.Tokenizer.Decode(generated));

For greedy/deterministic decoding, call model.SampleGreedy(logits) instead of Sample.

Smoke-test variant

A one-shot sanity check that loads a model, runs a single forward pass, and prints the top token:

using var model = ModelBase.Create(modelPath, backend);
var tokenIds = model.Tokenizer.Encode("Hello", addSpecial: true);
float[] logits = model.Forward(tokenIds.ToArray());

int topToken = model.SampleGreedy(logits);
Console.WriteLine($"vocab={model.Config.VocabSize}, tokens={tokenIds.Count}, topToken={topToken}");

DiffusionGemma — text diffusion

DiffusionGemma is a block text-diffusion model, not an autoregressive one. ModelBase.Create() still loads it (the GGUF architecture is diffusion-gemma / diffusion_gemma) and returns a DiffusionGemmaModel, but its Forward(int[]) intentionally throws — generation runs through DiffusionGemmaSampler, which iteratively denoises fixed-length canvas blocks over a [prompt | canvas] sequence. See the DiffusionGemma model card for the architecture.

using TensorSharp.Models;
using TensorSharp.Models.DiffusionGemma;
using TensorSharp.Runtime;

// Load a diffusion-gemma GGUF — the architecture is auto-detected.
using var model = (DiffusionGemmaModel)ModelBase.Create("diffusion-gemma.gguf", BackendType.GgmlCuda);

// Render the prompt with the model's chat template, then tokenize.
var messages = new List<ChatMessage> { new() { Role = "user", Content = "Write a haiku about winter." } };
string rendered = PromptRenderer.Render(model.Config.ChatTemplate, messages,
    addGenerationPrompt: true, architecture: model.Config.Architecture);
int[] promptTokens = model.Tokenizer.Encode(rendered, addSpecial: true).ToArray();

// Configure the EntropyBound denoising sampler.
var p = new DiffusionEbParams
{
    MaxDenoisingSteps = 48,   // refinement steps per canvas block
    Seed = 0,                 // deterministic
    MaxBlocks = 1,            // block-autoregressive canvas blocks
};

var sampler = new DiffusionGemmaSampler(model);

// Generate. The optional callback fires after every denoising step with
// (blockIndex, step, totalSteps, previewTokens) — useful for a live UI.
List<int> generated = sampler.Generate(promptTokens, p,
    (block, step, total, preview) => Console.Write($"\rblock {block + 1} step {step + 1}/{total}   "));

Console.WriteLine();
Console.WriteLine(model.Tokenizer.Decode(generated));

Key parameters on DiffusionEbParams: MaxDenoisingSteps (48), TMin/TMax temperature schedule (0.4 / 0.8), EntropyBound (0.1), StabilityThreshold / ConfidenceThreshold early-stop, Seed, and MaxBlocks. model.CanvasLength reports the per-block canvas size.

Qwen-Image-Edit — image editing

Qwen-Image-Edit takes a prompt + an input image and returns an edited image. The loaded qwen_image GGUF is only the MMDiT diffusion transformer; the model also pulls in two companion GGUFs resolved next to the DiT file (or via the TS_QWEN_IMAGE_VAE / TS_QWEN_IMAGE_TE / TS_QWEN_IMAGE_MMPROJ environment variables): the Qwen-Image VAE and the Qwen2.5-VL-7B text encoder. Like DiffusionGemma it is not an autoregressive text model — the autoregressive entry points throw and editing is driven through EditImage().

using TensorSharp.Models;
using TensorSharp.Models.QwenImage;
using TensorSharp.Runtime;

// Load the MMDiT GGUF (architecture = qwen_image). The VAE + Qwen2.5-VL
// companions are resolved from the same directory (or the TS_QWEN_IMAGE_* env vars).
using var model = (QwenImageModel)ModelBase.Create("qwen-image-edit-DiT-Q4_K_M.gguf", BackendType.GgmlCuda);

// Load the input image (PNG/JPEG decoded to an RgbImage).
RgbImage input = ImageIO.Load("input.png");

var p = new QwenImageParams
{
    Steps = 30,               // FlowMatch-Euler denoising steps
    CfgScale = 4.0f,          // true-CFG guidance; <= 1 disables the negative pass
    NegativePrompt = " ",     // used only when CfgScale > 1
    Seed = 0,
    TargetArea = 1024 * 1024, // ~1 MP; aspect follows the input (dims snapped to /16)
    // Width = 0, Height = 0, // pin an explicit output size to bypass the VRAM clamp
};

RgbImage output = model.EditImage("Make the sky a dramatic sunset.", input, p);
ImageIO.SavePng("edited.png", output);
Console.WriteLine($"Saved {output.Width}x{output.Height} edited image.");

For a live UI, set p.OnStep = (step, total, preview) => { … } and p.PreviewCount to receive decoded RGB snapshots of the partially denoised latent on throttled steps. RgbImage exposes Width, Height, and a planar/interleaved Pixels buffer; ImageIO provides Load, Decode(byte[]), EncodePng, SavePng, and resize helpers.

SamplingConfig

The sampling knobs map one-to-one to the CLI flags and API options. Defaults match Ollama.

Key types & interfaces

Other runtime contracts worth knowing: IModelArchitecture, IPromptRenderer, IOutputProtocolParser, IMultimodalInjector, IKvBlockCodec (with the built-in TurboQuantKvCodec), and IKVCachePolicy.