GPT Has No Idea What Words Mean

GPT does not understand language. It understands numbers.

Most people explain attention with metaphors. They use arrows or glowing connections. This is not how it works.

I traced the math by hand. I used actual matrix multiplications. Here is the reality of how a transformer works.

The process starts with embeddings. Every word gets a random coordinate. For example, the word "love" might start at [0.10, 0.30]. These numbers mean nothing at first.

Then attention happens.

Take the sentence: "The dog chased the cat because it was fast."

The model must decide if "it" refers to the dog or the cat. It has no grammar rules to follow. It only looks at previous words and assigns a score. It asks: "How much should I pay attention to you right now?"

This results in a weight table.

For the sequence [I, love, pizza]:

  • "love" might give 51% attention to "I" and 49% to itself.
  • "pizza" can see the whole picture. It spreads attention across all three words.

This is not a metaphor. The word "love" starts as one vector. After attention, it becomes a new vector. It physically changes its position in space. It absorbs part of the vector from "I".

The word "love" in the context of "I" is a different mathematical point than "love" standing alone. Context is just math blending vectors.

How does it learn? Through tiny shifts.

When the model predicts wrong, backpropagation nudges the weights. These changes are small. A weight might move from 0.400 to 0.397.

One tiny change of 0.009 can flip a prediction from wrong to right. GPT-3 did this 300 billion times.

This creates zero-shot generalization. The model learns that "You" and "I" are mathematically similar. It predicts "love" after "You" even if it never saw that specific pair. It does this because of geometric proximity.

The model never learns a rule. It learns a direction.

Every weight is a tiny compass needle. Training points these needles so that geometry becomes useful. There is no grammar book. There is only math.

Source: https://dev.to/pavan_kumar2004/gpt-has-no-idea-what-words-mean-thats-the-whole-point-16li

Optional learning community: https://t.me/GyaanSetuAi