Maksim Sekretov released a tiny, pure-JavaScript transformer that runs on Node.js and shows every scalar, weight and gradient as it trains. The repo tiny-language-model-neuro-js lets anyone start with a random model, feed it a prompt like “can human read ?”, watch it fail, then run a single-command training run and see the answer become correct.

Why most LLM demos hide the math

Typical language-model tutorials rely on TensorFlow or PyTorch. Those frameworks automatically build computation graphs and compute gradients behind the scenes, so learners see only high-level code and loss curves. The underlying operations—how each token becomes an embedding, how attention scores are calculated, how gradients flow back through every matrix—remain invisible.

What the JavaScript model does differently

Sekretov’s implementation strips everything down to a single dependency-free Node.js script. The training pipeline follows a straightforward sequence:

  • Tokenization
  • Embeddings
  • Causal transformer blocks
  • Multi-head self-attention
  • Feed-forward networks (FFN)
  • Language-model head and softmax
  • Cross-entropy loss
  • Backpropagation

Running node src/train.js --generalize --adaptive-teach launches the full training loop. Because the code is plain JavaScript, each tensor is a simple array, each matrix multiplication is an explicit loop, and every gradient sits next to its source weight.

From random guesses to correct answers

When the model starts, all parameters are random. Asking it “can human read ?” yields gibberish. After a brief pre-training phase and a round of supervised fine-tuning (SFT), the same prompt returns a sensible answer. The teacher signal—correct token sequences fed through the loss function—propagates back through embeddings, attention heads and FFN layers, updating each scalar in turn.

Tackling catastrophic forgetting

Early runs showed a classic problem: learning a new fact erased previously learned ones. Sekretov fixed this with rehearsal—re-presenting old examples while introducing new data. Training stops only when every target token reaches at least 95 % probability and survives ten consecutive checks. This simple loop demonstrates a core research challenge without any external libraries.

Who benefits and who might be left out

The project does not compete with commercial LLMs that run on billions of parameters and GPU clusters. That performance gap makes it unsuitable for production workloads, but it turns the model into a transparent teaching aid. Students, hobbyists and researchers who struggle to see inside a black-box framework now have a sandbox where the math is literal code.

What’s missing and where the project could go

Because the implementation is deliberately minimal, it keeps the code readable but limits scalability.

What to watch next

The repository is already public on GitHub.

Takeaway: By stripping a transformer down to pure JavaScript, Sekretov has turned a notoriously opaque piece of technology into a readable, editable script. The trade-off is speed, but the gain is insight—anyone can now watch a language model learn, forget, and relearn, scalar by scalar.