Everyone has an opinion on fine-tuning versus RAG. Scroll through any AI forum and you will find heated threads full of architecture diagrams and benchmark claims. Most of the people writing those comments have never trained a model on their own data or watched a RAG pipeline fail silently in production.

I spent months running experiments. Twelve of them, to be exact. I fine-tuned LLMs. I fine-tuned embedders. I built six different RAG configurations. The domain was financial prediction, specifically trying to forecast noisy market outcomes from messy historical data. I held myself to strict statistical standards because I wanted real answers, not blog post claims.

Most of the experiments failed. Those failures turned out to be far more useful than any lucky success.

The Hard Truth About Signal

Before getting into the weeds, here is the lesson that ties everything together. Fine-tuning and RAG are tools for changing what a model knows or what it sees. They are not magic wands that manufacture signal from thin air. If your underlying data does not contain a real, exploitable pattern, these techniques will not create one. They will just help you build a more convincing story around random noise.

In financial prediction, this trap is especially dangerous. Markets are noisy by design. When you strap a powerful LLM to historical price data and add retrieval or fine-tuning, you do not automatically get an edge. You get a more articulate way of rationalizing coin flips. If the signal is not there, the model becomes very good at lying to you. You need to check that first.

When Bigger Memorizes Instead of Learns

My first major mistake was assuming scale would fix everything. I tested a 14 billion parameter model against a 7 billion parameter model on exactly 777 training examples. The larger model achieved a noticeably better eval_loss. Its perplexity dropped. On paper, it was learning.

Then I looked at the win rate, the actual rate at which the model made correct predictions. The 14B model performed significantly worse than the 7B model. It had memorized the training noise. With fewer than 3,000 examples, the bigger model had enough capacity to overfit on spurious correlations and random wiggles in the data. It essentially built a lookup table of noise.

The 7B model, constrained by its smaller capacity, was forced to learn broader patterns. It could not afford to memorize every idiosyncrasy. If you are working with small datasets, start with smaller models. Scale is not free. It can actively hurt you when the data is thin.

Do Not Trust the Loss Curve

I learned to stop staring at loss curves. A model can improve its token-level cross-entropy while becoming worse at the actual business decision you care about. This happens because language modeling loss rewards predicting the next token accurately. In many domains, especially finance, the correct decision and the most probable next token are not the same thing.

I saw models that reproduced training prose beautifully but chose the wrong directional bet every time. The loss went down. The bankroll went down with it. Pick your evaluation metric based on the real-world task. If you are ranking documents, measure ranking quality. If you are predicting outcomes, measure decision accuracy. Never let eval_loss choose your model for you.

Fine-Tuning Shines Only on Foreign Vocabulary

I ran embedder fine-tuning trials on two different text types. The first used standard financial news and public filings. The fine-tuned embedder and the off-the-shelf version performed identically. The base model already knew this language. I was tuning on familiar ground.

The second dataset was full of private jargon, internal codenames, and domain-specific shorthand that never appeared on the open internet. Here, fine-tuning improved retrieval accuracy by 79 percent. The base model simply did not know what these terms meant. Fine-tuning taught it the local vocabulary.

This reframed the entire exercise for me. Fine-tuning is not about making a model smarter in some general sense. It is about teaching it a new vocabulary, a new format, or a house style. If your data looks like the internet, skip the fine-tune. If your data speaks a language the base model has never seen, fine-tuning becomes essential.

RAG Gives You Certainty, Not Truth

J'ai testé huit configurations RAG distinctes pour des tâches de prédiction. De manière générale, l'ajout de la récupération (retrieval) a modifié environ 30 % des décisions du modèle. Cela semble avoir un impact. Ça ne l'était pas. Ces changements n'étaient que du bruit pur. La précision globale ne s'est pas améliorée. Ce qui a changé, c'est la confiance du modèle. Le RAG a rendu le système plus assuré, a cité plus de sources et a produit des justifications plus longues. Tout en restant tout aussi erroné.

Cet excès de confiance est un risque produit. Un utilisateur voit des citations et suppose que le modèle a fait ses devoirs. En réalité, il ne faisait que des conjectures à l'apparence sophistiquée.

La leçon la plus douloureuse est venue du backtesting d'une variante de RAG. Elle affichait un profit annuel de 11 %. À première vue, cela ressemble à une stratégie gagnante. Mais son AUC, l'aire sous la courbe ROC et une mesure de la capacité de classification, était de 0,486. C'est pire qu'un lancer de pièce, qui se situe à 0,500. Le profit était un coup de chance lié à une période de marché spécifique, et non un avantage reproductible. Utiliser le P&L comme seule métrique est dangereux. Les marchés distribuent des séries de chance tout le temps. Vous avez besoin de métriques de compétence statistique pour distinguer la chance de la compétence.

Sachez ce que chaque outil fait réellement

Alors, où cela nous mène-t-il ? Utilisez le fine-tuning lorsque le modèle doit apprendre de nouveaux mots, des formats spécifiques ou un style distinctif. Utilisez le RAG lorsque le modèle a besoin d'accéder à des faits, des dépôts de code ou une mémoire institutionnelle qui réside en dehors de ses poids. N'utilisez aucun de ces outils pour chercher un signal dans des données qui n'en ont pas. Si le motif sous-jacent n'est pas là, la récupération et le fine-tuning ne feront que vous aider à habiller le bruit avec un costume plus élégant.

Le véritable goulot d'étranglement

L'infrastructure pour le fine-tuning et le RAG n'a jamais été aussi facile à mettre en place. Vous pouvez déployer un pipeline en un après-midi. La technique n'est plus le goulot d'étranglement. C'est l'évaluation qui l'est. La plupart des équipes sautent le travail statistique difficile et célèbrent plutôt des métriques de vanité. Elles livrent des systèmes qui ont l'air intelligents mais qui échouent silencieusement.

Effectuez des tests honnêtes avant de dépenser de l'argent. Remettez en question vos métriques. Vérifiez le surapprentissage (overfitting). Assurez-vous que le modèle est réellement meilleur,