I wanted to see if running the same question 51 times would make the answer more reliable. I took a local LLM, fed it a chunk of production code, and asked for a review. Then I did it again. And again. Fifty-one times total, using majority voting to pick the "best" response. The idea was simple: if the model stumbles on one run, maybe the wisdom of the crowd across 51 generations would cancel out the noise and surface the correct analysis. That is not what happened. The experiment showed that majority voting does not select for correctness. It selects for whatever the model is most stubborn about.
This distinction matters because majority voting has become a popular hack in LLM pipelines. The pattern is straightforward. You run the model several times on the same prompt, collect the outputs, and keep the answer that appears most frequently. In fields like medical imaging or spam detection, ensemble methods work because different models, or different views of the data, produce independent errors that actually do cancel out. Large language models are not independent voters. They are one system with one training history, one set of weights, and one universe of biases. When you ask the same model the same question fifty-one times, you are not convening a committee. You are taking a poll of the same respondent under slightly different moods.
The Persistence Problem
The core issue is that LLM errors are rarely random. They are patterns baked into the training data and the architecture. A model that misreads a specific Python decorator in one run is likely to misread it in the next. A model that hallucinates a security vulnerability because the variable name looks like a password will probably hallucinate it again on run forty-seven. The noise you are averaging out is usually surface-level variation in wording or formatting. The underlying reasoning often stays locked in place.
Consider a concrete example. Imagine a function that parses log files using a regular expression. The regex is strict and safe. But the string r'...' contains characters that, in a different context, might invite injection. Ask an LLM to review this. If the model has seen a thousand Stack Overflow posts warning against regex injection in log parsers, it may flag this safe snippet as a risk. Run it once, you get a false positive. Run it 51 times, and there is a good chance you get 51 false positives, or at least a strong majority. The majority vote now entrenches the hallucination. The model is persistent, so the "consensus" is persistent too.
This happens because temperature and sampling tricks do not change what the model knows. They only rearrange how it speaks. A high temperature might make the explanation chatty or terse. It might swap synonyms. It does not suddenly teach the model that the regex is actually harmless. The variations you are voting on are cosmetic. The error is structural.
What 51 Runs Reveal
When I spread those 51 outputs across my desk, the pattern was obvious. The model did not explore 51 different interpretations of the code. It rehearsed the same interpretation in slightly different voices. A handful of runs went off-script, suggesting edge-case fixes or noticing unrelated style issues. But the dominant cluster, the clear majority, kept returning to the same incorrect central claim. That claim was not correct. It was just familiar.
The mathematics of majority voting assumes independent Bernoulli trials. You need unrelated errors for the majority to outperform the individual. In my experiment, the errors were deeply related. They shared the same root cause: the model's training distribution overweights certain coding tropes. The majority vote, therefore, did not reduce error. It amplified the majority bias. It gave a false sense of certainty to a flawed analysis.
This is especially dangerous in code review because developers treat unanimous or near-unanimous AI output as authoritative. A single hesitant suggestion is easy to dismiss. A recommendation that holds steady across fifty-one runs feels like ground truth. It is not. It is ground loop.
Where Voting Actually Works
None of this means you should never run a model more than once. Majority voting can help in narrow situations where the task is shallow and the errors are truly random. Asking a model to pick between two syntactic formats, to choose a variable naming convention, or to extract a date string from a log line—these low-stakes tasks sometimes benefit from repeated sampling. The variation is genuine noise, and a quick vote cleans it up.
The trouble starts when the task requires reasoning about intent. Does this auth check belong here? Is this async call safe? Is this cache key collision actually exploitable? These questions demand an understanding of context, not just pattern matching. A model's pattern matcher is deterministic in its bias. It will reach for the most common answer from its training data, not the most accurate answer for your codebase.
Smarter Ways to Spend Your Compute
Fifty-one runs of a local model cost real time and electricity. There are better ways to invest that compute. If you want to improve reliability, diversity beats volume. Run two different models with