TensorSharp, a pure-.NET inference engine for GGUF language models, is now publicly available, letting .NET developers run large-language-model (LLM) inference without spinning up a Python server. The project claims performance comparable to the widely used llama.cpp while running on Windows, macOS and Linux and supporting CPU, CUDA, MLX, Metal and Vulkan back-ends.

Why native .NET matters

Most LLM runtimes are written in C/C++ or rely on Python wrappers that expose a separate process. For teams whose services already run on .NET, that extra layer adds containers, inter-process communication and a larger attack surface. Keeping inference inside the same runtime lets developers simplify CI/CD pipelines, shave latency from network hops, and cut the cost of maintaining a Python environment.

How TensorSharp is built

The author wrote the engine from scratch instead of reusing llama.cpp. The CPU backend is 100 % C#, so Windows and Linux run with no native dependencies. GPU support comes from existing graphics APIs: CUDA for Nvidia, MLX for Apple silicon, Metal for macOS GPUs, and Vulkan for cross-platform hardware. Modern tricks such as a paged key-value (KV) cache, continuous batching and speculative decoding for models like Qwen and Gemma sit in the core.

Features developers get out of the box

  • GGUF model compatibility – the same format used by recent open-source LLM releases.
  • Cross-platform operation – runs on Windows, macOS and Linux without code changes.
  • OpenAI- and Ollama-compatible HTTP APIs – drop-in replacement for existing client libraries.
  • Multimodal handling – can ingest images, video, audio and PDFs as part of a prompt.
  • Tool calling and structured output – supports function-calling patterns common in chat-based agents.

Performance versus llama.cpp

Benchmarks shared by the maintainer show mixed results:

  • Prefill and time-to-first-token (TTFT) – TensorSharp often beats llama.cpp, delivering lower latency for the initial response.
  • Decode throughput – usually similar to or slightly slower than llama.cpp.

The numbers suggest the pure C# implementation does not sacrifice speed in the most latency-sensitive phases, while staying competitive in raw token-per-second output.

Caveats to keep in mind

  • Real-world performance varies with model architecture, hardware configuration and memory layout; developers should run their own benchmarks before committing to production.

What to watch next

The project’s discussion channel on Telegram offers a place for early adopters to share results and request features.

Takeaway: TensorSharp gives .NET shops a way to embed LLM inference directly into their applications, removing the need for a separate Python stack and delivering latency comparable to the de-facto llama.cpp baseline. Production teams should validate performance on their target hardware, but the engine opens a clear path for native AI services inside the .NET ecosystem.