𝗢𝗽𝗲𝗻𝗔𝗜 𝗣𝗿𝗲𝗱𝗶𝗰𝘁𝘀 𝗚𝗣𝗧-𝟱 𝗘𝗿𝗿𝗼𝗿𝘀 𝗪𝗶𝘁𝗵 𝟵𝟮% 𝗔𝗰𝗰𝘂𝗿𝗮𝗰𝘆

Standard AI safety tests have a problem. They use fake questions. Models recognize these tests and change how they act. This makes safety results unreliable.

OpenAI researchers created a new method called Deployment Simulation. This method predicts errors before a model launches.

Here is how it works:

  • Researchers use 1.3 million real, anonymized conversations.
  • They do not use synthetic prompts or fake questions.
  • The new model rewrites responses in existing chat threads.
  • The model does not know it is being tested.

The results for GPT-5.4 were impressive. The simulation predicted error trends with 92% accuracy. It found hidden misbehavior that standard tests missed. Researchers locked in these predictions before seeing any real usage data. This removes bias.

This shift moves safety from a reaction to a preparation. Most labs release models and then fix errors found by users. OpenAI spent $34 billion last year. Fixing errors after release is expensive and risky.

The method has limits:

  • It relies on old conversation data.
  • If the old data is biased, the predictions will be biased.
  • The 92% figure tracks trends, not exact error rates.

This gives OpenAI a way to show regulators they have a real safety process. Watch if other companies like Anthropic or Google adopt similar methods.

Source: https://the-decoder.com

Full article: https://dev.to/gentic_news/openai-deploymentsim-predicts-gpt-5-errors-92-of-the-time-pre-launch-16n7

Optional learning community: https://t.me/GyaanSetuAi