Command Line (CLI)

TensorSharp.Cli is the console host for local prompts, multimodal experiments, prompt inspection, JSONL batch workflows, the interactive REPL, and built-in benchmarks. The binary lands in TensorSharp.Cli/bin/... after the build.

Quick start in ~30 seconds

Start at the TensorSharp repository root after installing the .NET 10 SDK for your platform. The verified quick start builds the native GGML backend and runs the repository's benchmark-verified gemma-4-E4B-it-Q8_0.gguf (7.48 GiB). The commands take about 30 seconds to copy and run; the model download and the first restore/build take longer and depend on your connection and machine. A native source build also needs Git, network access, CMake, and a working C++ toolchain.

Linux + NVIDIA

TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models/gemma-4-E4B-it-Q8_0.gguf
echo "Explain local inference in one paragraph." > prompt.txt
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models/gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

Windows PowerShell + NVIDIA

$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet build TensorSharp.slnx -c Release -p:TensorSharpSkipMlxNative=true
curl.exe --create-dirs --fail -L "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
Set-Content prompt.txt "Explain local inference in one paragraph."
dotnet run --project TensorSharp.Cli -c Release --no-build -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

Omit --input and the CLI falls back to its built-in What is 1+1? prompt; for your own one-shot text prompt, save the text to a file and pass --input. --prompt is only the Qwen-Image-Edit instruction.

Gemma 4 E4B on other native backends

On Apple Silicon, omit the CUDA environment assignment and use ggml_metal; on a supported Windows/Linux Vulkan GPU, request TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON instead and use ggml_vulkan; ggml_cpu runs the native CPU kernels and needs no GPU. The lower-memory gemma-4-E4B-it-Q4_K_M.gguf is in the same repository. Text needs no mmproj. Add the matching mmproj-gemma-4-E4B-it-Q8_0.gguf with --mmproj only for image, video, or audio input. See Getting Started for full platform syntax.

Examples

Run these source commands from the repository root. Replace the sample paths and choose a native backend that your build supports.

# Text inference (macOS)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_metal

# Text inference on Windows/Linux + NVIDIA GPU
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_cuda

# Text inference on any Vulkan GPU (AMD / Intel / NVIDIA)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --output result.txt \
    --max-tokens 200 --backend ggml_vulkan

# Multi-GPU host: list the visible Vulkan devices, then pick one by index
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- --list-gpus
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 200 \
    --backend ggml_vulkan --gpu-device 1

# Interactive turn-by-turn chat (REPL) with KV-cache reuse and slash commands
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal --interactive
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal -i \
    --system "You are a terse assistant." --temperature 0.7 --top-p 0.9 --think

# DSpark block speculative decoding (DeepSeek V4): --model is the FIRST shard of
# the split GGUF, --draft-model the separate DSpark drafter. Needs greedy sampling.
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <DeepSeek-V4-Flash-...-00001-of-00005.gguf> --backend ggml_cuda \
    --draft-model <DSpark-drafter.gguf> --input prompt.txt --max-tokens 200 --temperature 0

Multimodal

# Image inference (Gemma 3/4, Qwen 3.5-family)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --image photo.png --max-tokens 200 --backend ggml_metal

# Video inference (Gemma 4)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --video clip.mp4 --max-tokens 200 --backend ggml_metal

# Audio inference (Gemma 4)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --audio speech.wav --max-tokens 200 --backend ggml_metal

# PDF document Q&A (--input holds the question; scanned/image-only PDFs
# need a vision model + --mmproj, born-digital PDFs work with any model)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --pdf report.pdf --input question.txt \
    --max-tokens 400 --backend ggml_metal

Reasoning, tools & sampling

# Thinking / reasoning mode
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 400 --backend ggml_metal --think

# Tool calling
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 300 --backend ggml_metal \
    --tools tools.json

# With sampling parameters (the CLI defaults to greedy decoding)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --max-tokens 200 --backend ggml_metal \
    --temperature 0.7 --top-p 0.9 --top-k 40 --repeat-penalty 1.2 --seed 42

Image editing (Qwen-Image-Edit)

# Prompt + input image -> edited image. The VAE + Qwen2.5-VL text-encoder
# companions are resolved next to the DiT GGUF (or pass --qwen-image-vae /
# --qwen-image-vl / --qwen-image-mmproj).
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <qwen-image-edit-DiT.gguf> --image input.png \
    --prompt "Make the sky a dramatic sunset." --output edited.png \
    --backend ggml_cuda --diffusion-steps 30 --cfg 2.5 --diffusion-seed 0

DiffusionGemma & inspection

# DiffusionGemma text-diffusion generation
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <diffusion-gemma.gguf> --input prompt.txt --backend ggml_metal \
    --max-tokens 256 --diffusion-steps 48 --diffusion-seed 0

# Inspect the rendered prompt and tokenization without running inference
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input prompt.txt --dump-prompt

Batch & benchmarks

# Batch processing (JSONL)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --input-jsonl requests.jsonl \
    --output results.txt --backend ggml_metal

# Multi-turn chat simulation with KV-cache reuse (mirrors the web UI behavior)
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --multi-turn-jsonl chat.jsonl \
    --backend ggml_metal --max-tokens 200

# Throughput benchmark: best-of-N prefill and decode timing
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --model <model.gguf> --backend ggml_metal \
    --benchmark --bench-prefill 256 --bench-decode 128 --bench-runs 3

The JSONL format is one JSON object per line:

{"id": "q1", "messages": [{"role": "user", "content": "What is 2+3?"}], "max_tokens": 50}
{"id": "q2", "messages": [{"role": "user", "content": "Write a haiku."}], "max_tokens": 100, "temperature": 0.8}

Configuration file (--config)

Instead of a long command line, pass a JSON file with --config. The server reads the same format. Command-line options always win—file values are applied first, then anything you also pass on the command line overrides them, so one file can be reused across machines while you override just what differs. Repeat --config to layer files (later files win). Comments and trailing commas are allowed.

# Use the file, but override the backend for this run
dotnet run --project TensorSharp.Cli/TensorSharp.Cli.csproj -- \
    --config config/cli-basic.json --backend ggml_cpu

Keys are the same long option names below (with or without the leading --). A string/number becomes --key value, true becomes the bare switch --key, and an array becomes a repeated flag.

Variables. Define shared values once under "variables" and reference them with ${name} in any string value (an undefined name falls back to an environment variable of the same name). Declare as many roots as you need.

Auto-download. Any file option can be an object with a local path and one or more urls. If path is missing it downloads from the first working URL (mirrors tried in order), saves it there, and reuses it next time; progress prints to stderr, and an optional sha256 verifies the file.

{
  "variables": { "modelRoot": "C:/models", "hf": "https://huggingface.co" },
  "backend": "ggml_cuda",
  "max-tokens": 256,
  "temperature": 0.7,
  "model": {
    "path": "${modelRoot}/Qwen3.5-9B-Q8_0.gguf",
    "urls": [ "${hf}/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q8_0.gguf" ]
  }
}

Ready-to-use examples live in the repository's config/ folder (cli-basic.json, server-basic.json, variables.json, auto-download.json, qwen-image-edit.json) — each uses real, public, ungated URLs, so it works on a fresh machine. See config/README.md for the full reference.

Command-line options

Running the CLI with no arguments — or with --help (also -h, -?, /?) — prints the full parameter reference: every option with its description, default, range, and an example. It exits before any logging or model machinery starts. Unknown arguments are ignored, so copy option names carefully—a misspelling may otherwise appear to succeed.

Input / output

Runtime

Sampling

The CLI defaults to greedy decoding: temperature 0, top-k 0, top-p 1.0, min-p 0, penalties off, seed -1 (unlike the server, whose defaults match Ollama). Pass the flags below to opt into sampling; they also carry into the interactive REPL.

DiffusionGemma, benchmarks & logging

See the full reference, including --test, --test-templates, and chunked-prefill correctness checks, on the API Reference page.

Interactive REPL commands

Launch with --interactive / -i. Anything that does not start with / is a user turn; type /help for the list. The prompt header shows the current model, backend, architecture, context length, projector, conversation depth, and pending attachments. Press Ctrl+C while generating to interrupt; at the prompt to exit.

Conversation

Model & runtime

Sampling (live) & uploads (next turn)