Overview & Architecture
TensorSharp loads GGUF models locally and makes inference available to .NET applications. The command line, HTTP server and TensorAgent share engine capabilities, but they reach the engine through different host paths.
Start with your entry point
- Run a prompt or inspect a model: the CLI offers one-shot generation, a REPL, JSONL batches and benchmark commands.
- Use a browser or an existing client: the server provides a Web UI and OpenAI/Ollama-compatible APIs. Check the API reference for compatibility scope.
- Use a local application: TensorAgent combines chat, model management and personal knowledge. Validation differs by platform and release package.
- Embed inference in C#: begin with the minimal library example and reference the projects you need.
Two host routes, shared runtime capabilities
This map covers ordinary autoregressive text generation. The CLI has its own prompt and tool loop. The server and TensorAgent use the host-neutral Chat layer. Both routes can use Runtime's inference scheduling.
Command-line route
- CLI / REPLOwn prompt and interaction loop
- CliInferenceSessionSubmits sequences directly
- TensorSharp.RuntimeInferenceEngine
Referencing the Chat assembly does not route CLI requests through its pipeline.
Chat-host route
- Server / TensorAgentHTTP adapters or local app host
- TensorSharp.ChatSessions, generation and stream contract
- TensorSharp.RuntimeInferenceEngine
Chat has no ASP.NET Core dependency; Server adds HTTP around it.
Components used for model execution
- ModelsArchitectures and multimodal encoders
- Core and backendsTensors, operations and dedicated executors
- CPU / GPUImplementation depends on model and device
Diagram in words: CLI → CliInferenceSession → Runtime; Server / TensorAgent → Chat → Runtime. Runtime calls model implementations; models compute through Core, backends or dedicated executors. These are responsibilities, not one call stack traversed by every request. Source evidence →
Compose as needed: AgentHost supplies skills, tools, sandboxes and bounded agent loops to hosts; Runtime does not depend on it back. Knowledge is an independent personal store integrated by Chat, AgentHost and host layers.
How a text request runs
- Load the model: read GGUF metadata, select an architecture and bind weights according to backend capabilities. Generative models normally dispatch through
ModelBase.Create(). - Prepare inputs: render messages with the family's
ChatProtocol, template and tokenizer; encode attachments when the model supports their modalities. - Prefill: compute uncached prompt tokens and establish request state. Long prompts may be chunked; reuse depends on cache and conversation policies.
- Decode and stream: the scheduler advances sequences, the model computes candidate tokens, the sampler chooses outputs, and the host presents terminal text or API events.
Continuous batching can interleave requests, but cache layout varies by model: paged KV, native sequence slots or recurrent state. Speculation also depends on the model, placement and drafter. Scheduling and caches · Speculative decoding
Other workloads have dedicated paths: text diffusion and image/video models iterate a denoising pipeline rather than a token-by-token decode loop. Sentence encoders load through EmbeddingModel.Load(). See model guides and embeddings.
Find the next document by your question
| Your question | Next stop |
|---|---|
| Which projects should I reference? | C# package and project boundaries |
| Where do I extend a model? | Source directory map and architecture contracts |
| How does a model use the device? | Backends and dedicated executors |
| How do caches, batching and multiple GPUs fit? | Advanced features and multi-GPU / multi-node |
Check support and validation boundaries
An option being configurable does not validate every model, quantization, backend and device. Generic operations can fall back to CPU; dedicated executors have explicit requirements, and unsupported combinations may be refused at load.
Start with the model matrix and backend selector, then read the card for your workload. The retained capability and implementation notes preserve tool, safety, modality, cache and multi-GPU limits; source and release notes retain the previous distribution details.
Business value
For decision-makers evaluating local inference, the core trade is control and cost in exchange for running your own hardware.
Data privacy & compliance
Local inference and knowledge storage run on your hardware. Downloads, network-enabled tools and explicitly configured external companions follow the host configuration.
Predictable cost
No per-token API billing. Capacity is bounded by hardware you already budget for.
No vendor lock-in
Open GGUF models from Hugging Face, and an OpenAI/Ollama-compatible surface that existing tools already speak.
.NET-native integration
Embed inference inside existing C# services instead of bridging to an external runtime.
Scales on one box
Continuous batching serves many concurrent users from a single hosted model.
Migrate easily
Point existing OpenAI SDK code at http://localhost:5000/v1 and keep your application unchanged.
API compatibility still has endpoint and field limits, and local inference makes you responsible for models, memory and devices. Downloads, network-enabled tools and external companions follow host configuration. API compatibility · Tool and sandbox boundaries