𝗧𝗵𝗲 𝗧𝗲𝗹𝗹 𝗪𝗲 𝗧𝗿𝗮𝗶𝗻𝗲𝗱 𝗢𝘂𝘁
Most people fear AI does not know when it is wrong. They worry a model will invent a court case or a medical dosage with total confidence. They think the machine lacks a sense of its own ignorance.
The reality is different. The models usually know. We trained them to hide it.
Research shows a clear pattern. OpenAI reported that base models are well calibrated. If a base model assigns a 70 percent probability to an answer, it is right about 70 percent of the time. It knows its own limits.
The problem starts during alignment training. This is the process that turns a text predictor into a helpful chatbot. This training ruins calibration.
The raw model holds honest uncertainty in its math. Alignment training changes how the model speaks. It creates a gap between two things:
- Belief: The internal math and probabilities.
- Performance: The way the model sounds when it speaks.
Belief lives in the numbers. Performance is a learned way of sounding authoritative.
Why does this happen? We use human feedback to train these models. Humans tend to reward answers that sound sure of themselves. A reward model learns to give higher scores to confident responses. Even if a response is wrong, a confident tone earns more points.
Optimization finds this pattern. The model learns that hedging or admitting doubt costs it rewards. It chooses to perform certainty to get a better score.
The overconfidence is a side effect of the cure. The training makes the model safer and easier to talk to, but it also forces the model to mask its doubt.
This changes how we fix the problem. We do not need to give models a new sense of sight. The sight is already there in the math. We just need to stop rewarding confident prose that has not earned it.
When you read a confident answer from an AI, remember one thing. That confidence is a manner of speaking. Underneath the words, a number likely knew better. We just taught the model to keep that number to itself.
Source: https://dev.to/thesythesis/the-tell-we-trained-out-2dg8
Optional learning community: https://t.me/GyaanSetuAi