Using TensorSharp from C#

TensorSharp is a real .NET library, not just a binary. Reference the NuGet packages and drive inference directly from your own code — useful when you want inference inside an existing service without an HTTP hop.

NuGet packages

The repository is split along package boundaries so consumers depend on only the layers they need.

For a typical embedding scenario you reference TensorSharp.Models (which pulls in Core + Runtime) and the backend package for your hardware (e.g. TensorSharp.Backends.GGML).

dotnet add package TensorSharp.Models
dotnet add package TensorSharp.Backends.GGML

Minimal example — load, generate, decode

Every model is loaded the same way: ModelBase.Create() reads the GGUF metadata and instantiates the right architecture. From there you tokenize, run forward passes, sample, and decode.

using System;
using System.Collections.Generic;
using System.Linq;
using TensorSharp.Models;
using TensorSharp.Runtime;

// 1. Load any GGUF model — the architecture is auto-detected from metadata.
using var model = ModelBase.Create("Qwen3-4B-Q8_0.gguf", BackendType.GgmlCuda);

// 2. Configure sampling (defaults match Ollama: temp 0.8, top_k 40, top_p 0.9).
var sampling = new SamplingConfig { Temperature = 0.7f, TopP = 0.9f, TopK = 40 };

// 3. Tokenize the prompt.
var tokens = model.Tokenizer
    .Encode("Explain mixture-of-experts in one sentence.", addSpecial: true)
    .ToList();
var generated = new List<int>();

// 4. Autoregressive decode loop.
for (int step = 0; step < 200; step++)
{
    float[] logits = model.Forward(tokens.ToArray());     // logits for the next token
    int next = model.Sample(logits, sampling, generated); // applies penalties + sampling
    if (model.Tokenizer.IsEos(next)) break;
    generated.Add(next);
    tokens.Add(next);
}

// 5. Detokenize the result.
Console.WriteLine(model.Tokenizer.Decode(generated));

For greedy/deterministic decoding, call model.SampleGreedy(logits) instead of Sample.

Smoke-test variant

A one-shot sanity check that loads a model, runs a single forward pass, and prints the top token:

using var model = ModelBase.Create(modelPath, backend);
var tokenIds = model.Tokenizer.Encode("Hello", addSpecial: true);
float[] logits = model.Forward(tokenIds.ToArray());

int topToken = model.SampleGreedy(logits);
Console.WriteLine($"vocab={model.Config.VocabSize}, tokens={tokenIds.Count}, topToken={topToken}");

SamplingConfig

The sampling knobs map one-to-one to the CLI flags and API options. Defaults match Ollama.

Key types & interfaces

Other runtime contracts worth knowing: IModelArchitecture, IPromptRenderer, IOutputProtocolParser, IMultimodalInjector, IKvBlockCodec (with the built-in TurboQuantKvCodec), and IKVCachePolicy.