The port of Google’s Gemma-4 31B model to an AWS Inferentia2 inf2.24xlarge produced a perfect token-for-token match against the CPU reference—yet every generated sentence was gibberish. The gap between “matching” and “working” now serves as a warning for anyone trying to squeeze massive LLMs onto Amazon’s custom inference chips.

Why a token-by-token match isn’t enough

The developer compared each output token from the Inferentia device with the token produced by a CPU run of the model. The streams were identical, so the hardware appeared to have reproduced the reference implementation exactly. In reality, both streams fed a malformed prompt into a model stripped of its chat template and supplied with the wrong turn markers. The missing template sent the model into an infinite loop, spitting out nonsense. The hardware did its job—it reproduced a bug that existed in the reference code.

The lesson is simple: SEQ_MATCH (sequential token equality) does not equal correctness. If the reference implementation is broken, a faithful hardware replica inherits the same failure. Validation must go beyond token-level parity; it needs end-to-end functional checks with properly formatted inputs.

Buffers masquerading as parameters

During the load phase the model loader skipped a component called layer_scalar. The code registered this object as a buffer rather than a parameter in the PyTorch model definition. Buffers are static tensors that training does not update, and many loaders ignore them when converting to Neuron-compatible formats. Skipping it left scaling factors for several layers at their defaults, distorting the math across the entire network. No error was raised; the model compiled, and the inference pipeline ran, but the numerical results were off.

For anyone moving large models onto Inferentia, audit every non-parameter tensor. Even if a tensor is not meant to be learned, it may still be essential for correct forward-pass computation. Manually verifying buffer inclusion can prevent silent scale errors that are otherwise hard to diagnose.

Spot-instance volatility and the 39-minute compile

Running a 31-billion-parameter model on a spot instance looks cheap, but the savings come with unpredictable reclaim events. The developer’s compile time—about 39 minutes to translate the model into Neuron-compatible code—vanished when AWS reclaimed the instance. To survive interruptions they built a three-pronged safety net:

  • ModelBuilder kept memory usage within the 384 GB host limit, avoiding crashes that would force a restart.
  • Immediate S3 mirroring of both raw weight files and the compiled “neffs” (Neuron executable files) let a fresh instance pick up exactly where the previous one left off.
  • A multi-region poller scanned AWS regions for available spot capacity and launched a new instance as soon as one appeared.

These steps turned a fragile, single-point compile into a resilient pipeline that survives the churn of spot markets.

Sharding pitfalls with mixed attention layouts

Gemma-4 31B uses two attention configurations. Some layers employ four key-value (KV) heads, others a different count. Splitting the model evenly across eight parallel ranks fails when a layer’s KV head count does not divide cleanly. Trying to shard a 4-head layer across eight ranks would force each rank to handle half a head—a mathematical impossibility that triggers shape mismatches and runtime errors.

The fix was to replicate the globally-sharded layers (those with compatible head counts) across all ranks and only shard the “sliding” layers whose head counts allowed an even split. This hybrid strategy kept tensor-parallel efficiency while avoiding illegal division of KV heads, eliminating the tensor-parallelization errors that plagued earlier attempts.

Takeaway

Porting a giant LLM to Inferentia is more than a compile-and-run exercise. It demands rigorous functional testing beyond token equality, meticulous verification that every tensor—parameter or buffer—is correctly handled, and a deployment strategy that anticipates spot-instance reclamation. Finally, sharding must respect the model’s internal attention geometry; otherwise, the parallelism that promises speed becomes a source of silent failure.