Glossary & FAQ
New to local LLMs? Start here. Plain-language definitions of the terms used across this wiki, followed by the questions people ask most.
Glossary
| Term | What it means |
|---|---|
| LLM (large language model) | A neural network trained to predict text. You give it a prompt; it produces a continuation. |
| Inference | Running a trained model to get an answer (as opposed to training, which creates the model). TensorSharp is an inference engine. |
| GGUF | A single-file model format that packs the model's weights and metadata together. TensorSharp loads GGUF files, typically downloaded from Hugging Face. |
| Quantization | Compressing model weights to fewer bits (e.g. Q4_K_M ≈ 4-bit, Q8_0 ≈ 8-bit) so a model fits in less memory and runs faster, with a small quality trade-off. |
| Token | The unit a model reads and writes — roughly a word piece. "Tokens per second" measures speed; "context length" is how many tokens fit at once. |
| Tokenizer | Converts text to tokens and back. TensorSharp ships SentencePiece and BPE tokenizers. |
| Prefill vs decode | Prefill processes your whole prompt at once; decode generates the reply one token at a time. They have different performance characteristics. |
| KV cache | Stored attention "keys and values" from earlier tokens, so the model doesn't recompute them. It makes long contexts and multi-turn chats efficient. → paged KV |
| TTFT | "Time to first token" — how long before the answer starts streaming. KV-cache reuse reduces it on follow-up turns. |
| Backend | The hardware path that runs the math: CPU, NVIDIA GPU (CUDA), any Vulkan GPU (AMD/Intel/NVIDIA), or Apple Silicon GPU (Metal/MLX). → Backends |
| MoE (Mixture of Experts) | A model that routes each token to a few specialized sub-networks ("experts") instead of the whole network, giving large capacity at lower compute per token. |
| Multimodal | Able to take more than text — images, audio, or video — as input. → Multimodal |
| Thinking / reasoning | The model produces hidden step-by-step reasoning before its final answer; TensorSharp separates the two. → Thinking mode |
| Tool / function calling | The model can ask your application to run a function (e.g. "get_weather") and use the result. → Tool calling |
| Continuous batching | Serving many users at once by interleaving their requests on a single hosted model, instead of one at a time. → Deep dive |
| Speculative decoding | A small "draft" head guesses several tokens; the main model verifies them in one pass — faster output, identical result. → MTP |
| Block drafter / DSpark | A speculative drafter that proposes a whole block of tokens at once rather than one at a time, with a confidence head deciding how far to draft. DeepSeek V4 ships one as a separate GGUF. → DSpark |
| Tensor parallelism (TP) | Splitting one model across several GPUs so each holds a slice of every layer's weights — used when a model is too big for one card. Distinct from running several copies of a model. → Multi-GPU & Multi-Node |
| AllReduce | The collective step where every GPU in a parallel group sums its partial result with the others, so all of them end up holding the same value. TensorSharp runs it after each row-parallel projection. |
| Node / rank | In a distributed run, a node is one machine and a rank is one GPU's position in the parallel group. Global degree = local GPUs × nodes. |
| Projector (mmproj) | A companion file that lets a multimodal model understand images/audio. Place it next to the model or pass --mmproj. |
| Sampling | How the next token is chosen from the model's probabilities. Temperature, top-p, and top-k control creativity vs. determinism. → Sampling |
Frequently asked questions
Do I need a GPU?
No. TensorSharp runs on plain CPU (--backend cpu or the faster ggml_cpu). A GPU — NVIDIA (CUDA), any Vulkan-capable AMD/Intel/NVIDIA GPU, or Apple Silicon (Metal/MLX) — makes generation substantially faster, especially for larger models. See Backends.
Which model should I start with?
Start with Gemma 4 E4B Q8_0 (recommended public file: 7.48 GiB) — the verified quick-start model on TensorSharp's native-GGML fast path, with thinking, tools, and optional-projector image/video/audio input. A lower-memory small starter is Qwen3-4B Q4_K_M (2.33 GiB). See the E4B fast lane or model downloads.
Is my data private?
Yes — inference happens entirely on your machine. Prompts, documents, and images are not sent to any external service. That is the main reason organizations choose local inference. See Business value.
Can I keep using my existing OpenAI / Ollama code?
Yes. The server speaks both wire formats. Point an OpenAI client at http://localhost:5000/v1 (any API key) or an Ollama client at http://localhost:5000/api/…. See the HTTP API.
How do I serve many users at once?
Run TensorSharp.Server. Its continuous-batching engine interleaves concurrent requests against one hosted model and shares common prompt prefixes across them, which keeps throughput high. See Continuous batching.
How can I make it faster?
- Use a GPU backend (
ggml_cuda,ggml_vulkan, orggml_metal). - Pick a smaller or more aggressively quantized model.
- On supported models, enable speculative decoding with
--mtp-spec— or, on DeepSeek V4, load a DSpark block drafter with--draft-model. - Reuse sessions so the KV-cache prefix carries over between turns.
What does it cost?
There are no per-token fees — you run open models on hardware you control. The cost is the hardware and the electricity to run it. See Business value.
What platforms are supported?
Windows, Linux, and macOS (Apple Silicon), on .NET 10. Build from source today: the latest v3.0.5.0 release has no uploaded application archives. The repository has a cross-platform release workflow, so check the Releases page before assuming a prebuilt file exists. See Getting Started and platform binary status.
How is it licensed?
TensorSharp is authored by Zhongkai Fu and distributed under the license in the repository's LICENSE file (BSD-3-Clause). Each model you download has its own separate license from its publisher — check the model card on Hugging Face.
Where do I report issues or contribute?
On the GitHub repository.