vLLM beats SGLang on 64K-token prompts, but SGLang pulls ahead once the context hits 200K on an 8-GPU B300 server. The shift shows how decode-stage bottlenecks, not prefill work, dictate performance as token windows expand.
Why this benchmark matters
Long-context inference drives cost for chat assistants, code assistants, and any app that must keep hundreds of thousands of tokens in memory. Kimi-K3, a large-parameter model, is one of the first open-weight LLMs that can comfortably handle such windows, but the engine that runs it decides whether a request finishes in seconds or minutes.
vLLM and SGLang both promise high-throughput inference, yet they take opposite approaches to the decode stage. vLLM keeps the decode path simple, avoiding the extra synchronization that Decode Context Parallelism (DCP) introduces. SGLang spreads key-value (KV) cache reads across multiple GPUs using DCP, a technique that can amortize memory bandwidth at the cost of added communication.
The testbed
- Hardware: a single server with eight NVIDIA B300 GPUs, each with identical memory.
- Workloads: two context-length settings – 64 K tokens (the lower bound of “long context”) and 200 K tokens (the upper bound many research demos target).
- Metrics: total time to process a fixed batch of prompts; throughput derives from the time difference.
We held batch size, model weights, and memory-utilization targets constant. The only knob we changed between runs was the inference engine.
Numbers on the line
| Context | Engine | Time (s) | Relative speed |
|---|---|---|---|
| 64 K | vLLM | 100.5 | – |
| SGLang | 150.8 | vLLM ≈ 1.5× faster | |
| 200 K | vLLM | 295.2 | – |
| SGLang | 225.3 | SGLang ≈ 1.31× faster |
Throughput (tokens per second) fell dramatically for vLLM when the context grew: a 3.29× drop from 64 K to 200 K. SGLang’s throughput slipped only 1.25× over the same range.
Where the gap comes from
Both engines spend a similar amount of time on the prefill stage – loading the prompt into the KV cache. The divergence appears in the decode stage, where the model generates tokens one by one.
- 64 K tokens: inter-GPU communication dominates. vLLM’s single-GPU decode path sidesteps the extra synchronization required by DCP, letting it finish about 1.5× faster.
- 200 K tokens: the KV cache grows so large that reading it becomes the bottleneck. SGLang’s DCP, set to a size of 8, distributes those reads across all eight GPUs. The bandwidth gain outweighs the communication penalty, giving SGLang a clear lead.
A secondary, practical observation concerns memory pressure. Running either engine at a 0.95 memory-utilization target triggered out-of-memory (OOM) retries on the B300s. Dropping the target to 0.92 eliminated the retries and stabilized runtimes, at the expense of a modest increase in latency.
Who wins, who loses
- Developers with short-to-medium contexts (≤ 64 K tokens) gain more from vLLM’s lean decode path. Faster turn-around translates to lower cloud-compute bills and tighter user-experience loops.
- Teams building deep-analysis or research tools that need to keep hundreds of thousands of tokens in context should lean toward SGLang with DCP enabled. Its steadier throughput reduces the risk of time-outs and keeps GPU utilization higher as memory demands rise.
- Hardware planners see that raw GPU count alone does not guarantee linear scaling. When KV bandwidth becomes the choke point, architectures that can parallelize cache reads – either via DCP or future memory-subsystem upgrades – will extract more value from the same silicon.
Counter-point: can vLLM close the gap?
Until new data appear, the current numbers stand as the best public comparison.
What to watch next
- Larger context windows will stress KV bandwidth even more, potentially widening SGLang’s lead.
Bottom line
When you need to serve long-context requests on a B300-based cluster, pick the engine that matches your token window. For 64 K-token workloads, vLLM delivers roughly 1.5× faster inference. Push past 200 K tokens, and SGLang’s DCP-driven decode becomes the more efficient choice, with its throughput dropping 1.25× compared with vLLM’s 3.29× drop. Adjust memory utilization to 0.92 to avoid OOM retries, and remember that the “best” engine is context-dependent, not a one-size-fits-all solution.