Beyond Transformers: The Startups Redefining the Next Era of LLMs

The era of the transformer is facing a fundamental bottleneck as the computational demands of large language models (LLMs) reach a breaking point. While the 2017 "Attention Is All You Need" paper laid the foundation for the current AI boom, a new generation of startups is now racing to replace or reinvent the core architecture to achieve true efficiency.

The Transformer Bottleneck: Why Dense Attention is Failing

For nearly a decade, the transformer has been the engine of the AI industry, powered by a mechanism known as "dense attention." This process compares every token in a block of text with every other token via massive-scale multiplication. While incredibly accurate at capturing semantic meaning, the mathematical complexity scales poorly.

For example, a 10,000-word document can require a transformer to perform roughly 50 million multiplications. This quadratic growth in computation leads to two massive challenges: skyrocketing energy consumption and limited context windows. With OpenAI reportedly set to spend $50 billion on computing this year and data center electricity demands expected to double by 2030, the industry is hitting a wall of both cost and physics.

Rethinking Attention: The Rise of Sparse Mechanisms

To solve the efficiency crisis, several startups are attempting to move away from dense attention toward "sparse attention." Instead of calculating every possible word pairing, sparse attention focuses on only the most relevant connections, significantly reducing the computational load.

Miami-based startup Subquadratic is a leader in this space with its SubQ model. Unlike previous sparse mechanisms that struggled to maintain accuracy, Subquadratic claims its technology rivals mainstream LLMs in specialized tasks like coding and search. Their approach involves a dynamic system that identifies which words actually matter for a given piece of text on the fly, rather than wasting cycles on irrelevant tokens.

Power Retention: Moving Toward Rolling Summaries

Another approach involves moving away from the attention mechanism entirely. San Francisco-based Manifest AI is pioneering a technique called "power retention." While sparse attention models like SubQ still try to retain a rough picture of the entire context window, power retention works by providing the model with a rolling summary of its memory.

As new information enters the context window, the model identifies and drops less relevant data, ensuring the computational load remains manageable regardless of input size. Manifest AI has already demonstrated the viability of this approach by:

  • Converting the open-source StarCoder model into PowerCoder using power retention.
  • Releasing Brumby, a model they claim rivals Alibaba’s popular Qwen open-source models.

The ability to adapt existing transformer models into power retention models with minimal retraining suggests that the transition to "LLMs+"—the next generation of models—could happen much faster than anticipated.

Key Takeaways

  • The Transformer Limit: The quadratic scaling of dense attention makes current LLMs incredibly expensive to run and difficult to scale for massive datasets or long-form reasoning.
  • Sparse vs. Retention: Startups are attacking the problem from two sides: Subquadratic is optimizing attention via sparse calculations (SubQ), while Manifest AI is replacing it with rolling summaries (Power Retention).
  • Economic Imperative: As AI compute costs reach tens of billions of dollars, the winner of the next AI era will likely be the one who solves for efficiency rather than just raw scale.