Overview & Architecture

TensorSharp loads GGUF models locally and makes inference available to .NET applications. The command line, HTTP server and TensorAgent share engine capabilities, but they reach the engine through different host paths.

Start with your entry point

Two host routes, shared runtime capabilities

This map covers ordinary autoregressive text generation. The CLI has its own prompt and tool loop. The server and TensorAgent use the host-neutral Chat layer. Both routes can use Runtime's inference scheduling.

Hosts handle interaction; Runtime schedules model execution

Command-line route

  1. CLI / REPLOwn prompt and interaction loop
  2. CliInferenceSessionSubmits sequences directly
  3. TensorSharp.RuntimeInferenceEngine

Referencing the Chat assembly does not route CLI requests through its pipeline.

Chat-host route

  1. Server / TensorAgentHTTP adapters or local app host
  2. TensorSharp.ChatSessions, generation and stream contract
  3. TensorSharp.RuntimeInferenceEngine

Chat has no ASP.NET Core dependency; Server adds HTTP around it.

Components used for model execution

  1. ModelsArchitectures and multimodal encoders
  2. Core and backendsTensors, operations and dedicated executors
  3. CPU / GPUImplementation depends on model and device

Diagram in words: CLI → CliInferenceSession → Runtime; Server / TensorAgent → Chat → Runtime. Runtime calls model implementations; models compute through Core, backends or dedicated executors. These are responsibilities, not one call stack traversed by every request. Source evidence →

Compose as needed: AgentHost supplies skills, tools, sandboxes and bounded agent loops to hosts; Runtime does not depend on it back. Knowledge is an independent personal store integrated by Chat, AgentHost and host layers.

How a text request runs

  1. Load the model: read GGUF metadata, select an architecture and bind weights according to backend capabilities. Generative models normally dispatch through ModelBase.Create().
  2. Prepare inputs: render messages with the family's ChatProtocol, template and tokenizer; encode attachments when the model supports their modalities.
  3. Prefill: compute uncached prompt tokens and establish request state. Long prompts may be chunked; reuse depends on cache and conversation policies.
  4. Decode and stream: the scheduler advances sequences, the model computes candidate tokens, the sampler chooses outputs, and the host presents terminal text or API events.

Continuous batching can interleave requests, but cache layout varies by model: paged KV, native sequence slots or recurrent state. Speculation also depends on the model, placement and drafter. Scheduling and caches · Speculative decoding

Other workloads have dedicated paths: text diffusion and image/video models iterate a denoising pipeline rather than a token-by-token decode loop. Sentence encoders load through EmbeddingModel.Load(). See model guides and embeddings.

Find the next document by your question

Your questionNext stop
Which projects should I reference?C# package and project boundaries
Where do I extend a model?Source directory map and architecture contracts
How does a model use the device?Backends and dedicated executors
How do caches, batching and multiple GPUs fit?Advanced features and multi-GPU / multi-node

Check support and validation boundaries

An option being configurable does not validate every model, quantization, backend and device. Generic operations can fall back to CPU; dedicated executors have explicit requirements, and unsupported combinations may be refused at load.

Start with the model matrix and backend selector, then read the card for your workload. The retained capability and implementation notes preserve tool, safety, modality, cache and multi-GPU limits; source and release notes retain the previous distribution details.

Business value

For decision-makers evaluating local inference, the core trade is control and cost in exchange for running your own hardware.

🔒

Data privacy & compliance

Local inference and knowledge storage run on your hardware. Downloads, network-enabled tools and explicitly configured external companions follow the host configuration.

💸

Predictable cost

No per-token API billing. Capacity is bounded by hardware you already budget for.

🧩

No vendor lock-in

Open GGUF models from Hugging Face, and an OpenAI/Ollama-compatible surface that existing tools already speak.

🏗️

.NET-native integration

Embed inference inside existing C# services instead of bridging to an external runtime.

📈

Scales on one box

Continuous batching serves many concurrent users from a single hosted model.

🔁

Migrate easily

Point existing OpenAI SDK code at http://localhost:5000/v1 and keep your application unchanged.

API compatibility still has endpoint and field limits, and local inference makes you responsible for models, memory and devices. Downloads, network-enabled tools and external companions follow host configuration. API compatibility · Tool and sandbox boundaries