Nvidia dropped Nemotron 3.5 Lightning on August 11. The 30-billion-parameter mixture-of-experts model runs with only 3 billion active parameters, and its companion NeMo Switchyard routing library has already sparked a debate over inference costs and cache efficiency. The way Switchyard shuttles requests between models can trash prompt caches and potentially double the bill for a single inference session.

A model built for open-weight agents

Nemotron 3.5 Lightning targets AI agents that call tools, validate results, and keep a reasoning trace. Three technical choices set it apart:

  • Hybrid architecture – It fuses a Mamba-2 state-space core with a conventional Transformer, proving that non-Transformer designs can compete at this scale.
  • 4-bit floating-point quantization – Shipping in low-precision lets the 30 B model churn out roughly 100 tokens per second on an M5 Max MacBook Pro, a speed a full-precision model would struggle to match.
  • Full openness – Nvidia released the weight files and the complete training recipe, a rarity for a model aimed at high-performance inference.

Early users say the model can “over-think,” spitting out long reasoning chains that sometimes err. Developers get a powerful, openly available agent executor but must prompt carefully to avoid needless verbosity.

Why the routing layer matters

NeMo Switchyard is Nvidia’s library for dynamically routing a request to the “best” model among a pool of candidates. In theory, a router can boost answer quality by picking a specialist model for each query. In practice, the routing decision clashes with prompt-caching mechanisms that many inference pipelines rely on.

  • Prompt caches store token embeddings of the initial prompt so later generations can reuse them without recomputing the “prefill” step.
  • Routing swaps move a request from Model A to Model B after the first few tokens, forcing the system to discard the cached prompt and recompute it from scratch for the new model.

Since most AI spend goes to reading the cached prompt rather than generating new tokens, a single routing hop can effectively double the cost of a request.

The cost tug-of-war

What’s on the horizon

The debate arrives as Nvidia’s open-weight strategy meets direct competition. On August 15, Qwen 3.8-27B will launch, and early benchmarks suggest it can hold its own against Nemotron 3.5 Lightning in local development scenarios.

Nvidia’s pattern—open weights, open training recipes, and an open routing layer—looks designed to make its GPUs the default hardware for anyone running these models locally. By exposing the software stack, Nvidia hopes to lock developers into its ecosystem, even if the routing logic adds hidden cost penalties.

Takeaway

If you plan to run Nemotron 3.5 Lightning on your own machine, ask not only “how fast can it generate?” but also “how often will Switchyard force me to discard my prompt cache?” That answer will determine whether the model’s open-weight promise becomes a real-world saving or an expensive experiment. As alternatives like Qwen appear, the balance between routing flexibility and cache-driven cost efficiency will decide which toolkit—and ultimately which hardware platform—wins.