A Four-Type Framework for LLM Wiki

Knowledge alone does not create judgment.

I built an LLM wiki with over 100 pages. It worked well until I tried to use it as a programming tutor.

The AI knew every definition. When a student said, "I do not understand Promises," the AI gave a textbook definition. This was a bad response.

A real tutor would ask: "Do you understand callbacks first?" or "What have you tried?"

The AI had facts but zero judgment. I realized my wiki only stored one type of knowledge.

Most LLM wikis store declarative knowledge. These are facts, definitions, and summaries. They answer "What is this?"

Expertise requires more. There are four types of knowledge you need:

• Declarative: Facts and definitions. • Procedural: How to reason. This is the sequence of decisions an expert makes. • Experiential: How it works in real life. This includes seeing mistakes and how experts handle them. • Interaction: How to guide others. This is knowing when to ask a question and when to stay silent.

Research shows a huge gap between knowing and doing. In one study of teaching assistants, 75% of them gave answers instead of using Socratic questioning. They knew the rules, but they could not execute them under pressure.

To fix this, you need two operations in your system:

  1. Ingest: This looks for facts to create declarative knowledge.
  2. Mine: This looks for decisions to create procedural, experiential, and interaction knowledge.

You cannot find a reasoning path by looking for facts. You must look for decisions. You must ask: What did the expert choose? When did they choose it? What happened next?

The next generation of AI will not win by having more facts. It will win by having better judgment. Judgment is not a knowledge problem. It is a knowledge-type problem.

If you build AI that only stores facts, you are only building a textbook. If you want to build an expert, you must mine for decisions.

Source: https://dev.to/rongrong/a-four-type-framework-for-llm-wiki-by-karpathy-5f1n

Optional learning community: https://t.me/GyaanSetuAi