Article: SkewAdam slashes Mixture-of-Experts training memory by more than 60% and delivers a noticeable jump in accuracy, letting developers fit a 6.78 billion-parameter MoE model on a single 40 GB GPU.

Why MoE training hits a memory wall

Mixture-of-Experts architectures route each input through a small subset of “expert” sub-networks while keeping a dense backbone. The benefit is a model that can scale to billions of parameters without a proportional rise in compute. In practice, the optimizer—specifically the AdamW variant most teams use—eats up the bulk of GPU RAM. For a 6.78 B-parameter model, the optimizer’s momentum and variance tensors alone demand about 50 GB. Add activations and model weights, and peak memory tops 81.4 GB, forcing multi-GPU setups or slower training schedules.

What SkewAdam does differently

SkewAdam does not invent a new learning rule. It reshuffles where Adam’s moments live:

  • The dense backbone keeps full-precision momentum, preserving the smooth updates dense layers need.
  • The expert bank stores a factored approximation of the variance, cutting the data kept for each expert.
  • The router, which decides which experts to activate, retains an exact second-moment estimate.

By treating these three components as separate “tiers,” the optimizer drops redundant precision where it matters less and keeps it where it matters most.

Measurable impact

Running the same 6.78 B-parameter MoE model with SkewAdam yields:

  • Peak GPU memory: 31.3 GB (down from 81.4 GB)
  • Optimizer-state footprint: 1.29 GB (down from 50 GB)

The reduced state fits comfortably inside a single 40 GB accelerator.

Accuracy gains, not just savings

Memory cuts often hurt model quality, but SkewAdam improves perplexity—a standard language-model metric—from 126.8 (AdamW) to 108.4. It also outperforms Muon (120.2) and Lion (393.7) on the same task. The authors say the boost comes from preserving momentum across all three tiers; methods that drop momentum entirely, like Adafactor, lag behind.

Open questions

SkewAdam’s results are demonstrated on a 6.78 B-parameter model. It remains unclear whether the same memory-state ratios hold for models an order of magnitude larger or for tasks beyond language modelling.

What to watch

  • Benchmarks on larger MoE configurations (tens of billions of parameters).
  • Adoption by open-source frameworks and inclusion in popular training scripts.
  • Community feedback on hidden trade-offs, such as convergence speed or stability under different learning-rate schedules.

Source: developer post detailing the optimizer’s design and empirical results.